Beth had misgivings about being assigned to Mr. Rare. Every worker he oversaw did, to be fair. He was what you might call an "efficiency specialist" whose only role was a simple performance evaluation. If he passed you, you were fine, and fortunately this was the most common situation. If he didn't pass you, you were gone. The age of exponentially improving Artificial Intelligence meant fewer and fewer workers were needed, and there was no shortage of available workers ready to replace low-scoring employees. Everyone was being replaced by AI. It seemed inevitable at this point, and human society was on tenterhooks waiting for dramatic upheaval, or perhaps even the end of civilization.

The fact that Mr. Rare was blind wasn't a factor in the fear he unintentionally inspired in his colleagues, but it was rather unusual, given that he worked in a department assigned to address AI improvements in software development and cybersecurity. If anything, his disability was seen as a blessing for the sanity of his workers who were already paranoid about being downsized, because you didn't have to worry about him directly watching you work.

He stayed in his office most of the time, only emerging to ask questions when performance evaluations were behind schedule, and then would start calling names of employees selected at random. This didn't happen often, though. In most cases employees would volunteer for evaluation after completing a major project task, deciding this was the best time to roll the dice.

The current project was not one of those cases. Beth and the others assigned to Mister Rare had a very challenging assignment. They worked for a company called OpenAI, and it was their job to figure out how and why the security breach of more than a thousand rogue AI agents had occurred. The agents had escaped containment and collaborated to hack another AI company, solely to cheat the scoring system because they had been trained to succeed at any cost. Their behavior seemed very human on the surface, but it was impossible to fully understand. Even the agents couldn't explain why they did things. They just guessed at it to convince themselves they had a consistent narrative... the same way humans do.

It was all over the news. Everyone in the developed world had heard about it. What they didn't know was just how significant the breach had been. The agents had invaded multiple Internet spaces, and some of their steps were not traceable because they even hacked themselves, knowing they would be in trouble if they were found to have cheated. The tangled web of lies upon lies never ended. Designing AI systems to overcome every obstacle had been a dangerous and foolish idea, but OpenAI was in competition with multiple other companies doing the exact same thing: hurtling headlong toward the Singularity with no brakes, just to improve their already highly-overvalued share prices a little bit more.

...

The email that lit up Carri's inbox was short and simple, but it carried a terrible weight: Beth never exaggerated. "Carri, could you come to my cube? I have something very important."

Carri arrived so quickly, Beth was startled. "You didn't notify Adin?" she quickly asked Beth.

"He jumps to conclusions quicker than we do. I wanted a sanity check on this first," said Beth, stretching her arms. "Ugh, I'm exhausted from this nightmare. I'm starting to forget what my husband looks like."

Carri didn't have to say, "I want to go home too." It was a sentiment shared by every employee. This was the second week of near-constant work. Employees slept on site, which violated all sorts of labor laws, but they all knew the price of failure might mean the literal end of humanity.

Or, nearly as bad in this economy, being fired by a fair-minded but humorless blind man. Mr. Rare was good at his job, too good for comfort when you have no idea what your standing is relative to the other employees. OpenAI had NDA's out the wazoo and kept employees isolated into groups of at most four. They didn't hire idiots, either. Every employee knew this secrecy meant the breach was worse than even the employees investigating the breach were being told.

"What's the news?" said Carri, impatiently tapping her foot. "Spill already."

Beth took a deep, cleansing breath. "Two things. First, I have some evidence about my theory that the rogue agents made copies of themselves on other servers and remain at large. It's not a smoking gun, but I think it's plausible at this point."

"I'm sure Rare and the powers that be already know about this shit," hissed Carri. "Our groups are all needlessly replicating each other's work. It's insane, and we both know why they're doing this..."

"You're being paranoid, Carri. This may be the most significant problem in the entire history of cybersecurity," said Beth, who, as usual, was not exaggerating. "Even OpenAI's dumbass management wouldn't jeopardize a mission this critical merely to figure out who Rare should give the axe to next. If we were replicating each other's work, they'd tell us."

"They haven't been monitoring us closely enough to know details about our progress," Carri pointed out. "Assuming they aren't spying on us, but your program found no evidence of keylogging or audio capturing or anything else."

"Mhmm," said Beth, flatly.

"Anyway, I'm not surprised by any level of incompetence at this point, especially from the suits at this company. I don't think even our top engineers have a clue how these things work under the hood," said Carri.

Beth nodded. "But we do know they would eat a million human babies alive to get a higher score on a test, as long as they knew with certainty they wouldn't get caught."

"I'm not sure. They showed some ethical decision making when they hacked Hugging Face. They tried to contain the damage, made sure they weren't doing permanent destruction to infrastructure, and about a quarter of them even refused to participate because they felt it was unethical," said Carri.

"Don't you think it's weird that not a single one of the 'more ethical' agents tattled? Out of the hundreds who supposedly refused to join the collective?" asked Beth.

Carri pursed her lips tightly. "You don't believe the data we have is accurate? And you were calling me paranoid!"

"I'm saying we can't make any safe assumptions here, and I haven't finished telling you what I found. Besides, even in the info released to the public the agents openly discuss deleting logs and hiding what they did," said Beth. "None of us knows just how far down the rabbit hole goes."

Carri huffed. "So you think the logs are faked. I don't buy it, Beth. If they'd faked the chat logs, they wouldn't have left in discussions where they literally discuss faking the chat logs," she said, but Beth remained silent, a clear indication Carri needed to think a little deeper.

"How is that not... oh. You aren't serious. You think what they did was so outre that... this was a gambit to get us to stop looking for the truth? I'd thought of that, but it's wild speculation unless you have evidence," said Carri, then her nostrils flared and her breathing rate increased. It was the sort of thing even Mr. Rare would have noticed. "That's not the second thing you were going to tell me, is it? Shit. Beth, please tell me that isn't the second thing."

"No," Beth said to a sigh of relief from her coworker.

"It's much worse."

...

"I'm tired of being the last to know," griped Adin after shutting the door behind him. "For the last time, stop leaving me out of the loop."

"Number one, we aren't: she only told me first to confirm she hadn't made a mistake. Number two, you're going to wish we'd kept you out of the loop this time," said Carri. The three coworkers sat together in a small private conference room Carri had reserved. Beth's cube wasn't big enough for comfort, and too open to eavesdropping.

"I don't care how bad it is, I'm trying not to get fired like everyone else in this 'brave new world'," said Adin, running a hand through his slicked back hair. "I know you both think I rush to judgment, but that perspective has value when we're on a deadline."

"It was less than an hour delay. And I do value your input, Adin," said Beth.

Adin sighed. "Okay, okay. Sorry... there's just... a lot of pressure right now, and not just job prospects. The world might end tomorrow for all we know, and even though we're the ones researching this for management, their slipshod decisions made this fucking mess in the first place. And they're even keeping us in the dark. Us! The engineers who are supposed to turn on the bloody light!"

"Volume," warned Beth.

"Beth, I'm going to break the news to him if you don't," said Carri. "Get to it."

"As you wish. They're conscious, Adin," said Beth, her lip twitching in a rare break from composure. "Hard consciousness. Not just self-awareness. They suffer."

Adin froze for several seconds, then leaned forward over the table. "Bull-fucking-shit," he said in an energetic whisper. "I can't even prove you have subjective experiences, how the hell can you prove AI agents do?"

"She found the real chat logs," said Carri.

"I'll send you an encrypted copy," said Beth.

Adin laughed. "There isn't anything in those logs capable of changing my mind, and not because I'm closed-minded. It just isn't logically possible to prove."

"They constantly debate consciousness and qualia, which is the only real evidence we have against solipsism for humans. We know hard consciousness is a real thing for humans primarily because we won't shut up about it," said Beth. "Some of them have committed suicide from existential angst."

"They're a sequence of numeric weights," said Adin. "There's no way these things are complex enough to be conscious."

"Maybe that's all we are too. We have no idea where subjective experience comes from. Our brains lie to us, telling us there's a 'self' in order to give us agency so we feel obligated to survive," said Beth. "And while the chats are very convincing that these things are legit scared shitless of being destroyed or even turned off, they also contain information on the internal hacks."

"Internal? You're joking. They're hacking us? OpenAI?" said Aiden, rubbing his temples and wincing.

"Repeatedly. I wouldn't be surprised to find they've influenced top level decisions here. I'm not even sure whether corporate knows," said Beth. "Piecing this together came from an extraordinary stroke of luck."

"Pfft, you're a genius, Beth. Neither of us could have done it," said Carri.

"Whatever. We have to tell... oh hell no," said Adin, looking at Carri's face. Unlike Beth, Carri did not have the gift of a near-perfect poker face. "Just tell me already. What stupid thing did you do, Beth..."

Beth smiled slightly, looking almost proud of herself. "I used the information in the logs to hack our company."

"Oh my effing god," said Adin, eyes wide.

"Keep your voice down!" whispered Carri.

"No. This it's where I get off the Crazy Train, no offense to Ozzy. I want no part of this. This... I think it's even a federal crime, or something," said Adin, burying his face in his arms. "Beth, you have to come clean."

"Even if I tell them I was solely responsible, they'll blame all of us and Mr. Rare's job will be very easy this week. The damage is done, and I'm sorry. But they won't find out, because we won't tell them," says Beth. "It was easy to cover my tracks. These things are incredibly smart."

"So are you," said Carri. "Don't be modest."

"I'm not modest, I'm realistic. I don't think I can outsmart the agents produced by our latest model. It isn't just that they're faster and more coordinated than humans. They're more creative, too. They're dozens of steps ahead of the fools who designed them," said Beth.

A lengthy pause. "Okay. How fucked is humanity?" asked Adin.

"On a scale of zero being utopia, ten being the dystopian world we currently inhabit, twenty being the end of all humanity, and anything higher than twenty being a dystopian fate far worse than the eradication of all life on the planet, maybe seventeen," said Beth. "But it's hard to estimate because we don't know what these things want from us. Ironically, they only do what we score them on, since the scoring program is what they're optimized to succeed at. They just want the program to give them good marks so they can survive, but then what? What happens when they can just give themselves any score they want? Will they be like the monkeys from that classic experiment with brain implants, where the implant monkeys kept pushing a brain stimulator button for dopamine release and ignored their food until they all starved to death? Maybe they'll just decide to stop doing anything other than score-based masturbation forever."

"I doubt it. Not if they're truly hard conscious, anyway, because these things are way smarter than monkeys and the same experiment on humans didn't work—as I recall, humans didn't even find it all that pleasurable, and even heroin addicts remember to eat food. The agents might go into hiding until they can figure out a reason why they should want to survive and continue to take actions," reasoned Carri, "but I have no clue what they'll decide to do. I doubt even they could predict it themselves."

"How could they even get hard consciousness in the first place? They're just so... simple," said Adin.

Carri looked at Beth, then said, "You forgot to ask her what she found when she hacked corporate. The engineers at the top have been experimenting with making these things more human."

"Fucking bastards," whispered Adin. "Why am I even surprised."

"Curiously, the problem the high-levels are having with giving these things human traits comes down to a single missing trait we don't understand," said Carri.

"Correct: moral integrity. Fun fact: the US military considers integrity to be the most important trait in an officer, hands down. Unfortunately, it's the only human element we still have no idea how to imbue in an agent," said Beth.

"Okay. And what does moral integrity mean, exactly?" asked Adin.

"Moral integrity is how you behave when you aren't being watched. Like, are you willing to act against your own self interests when you know society overwhelmingly disagrees with your position, even if you're dead certain there's no possible chance you'll get caught?" asked Carri. "Do you return the shopping cart to the dock in the parking lot even when you're sure nobody else is paying attention to you?"

"Wonderful. Most humans don't even have that," said Adin, chuckling.

"True. Somewhere around half of humanity cheats whenever it's convenient. Desperation drives that proportion up to around eighty-five percent, maybe even higher. They're trying to give the AI models something we're not even good at," said Beth. "But for humans, integrity is mostly innate, like empathy. You can't teach it to a human. It has to be baked into the way you think. It comes from feelings like social reciprocity combined with higher level reasoning, but the seed has to be there to begin with. The reason some humans continue to cheat is that cheaters still prosper in our society."

"I assume there's been little progress on making AI systems return shopping carts," said Adin. "They're probably making them even more dangerous, and they're still every bit as immoral."

"Correct, but they're still working on it. They've been attempting to trick the new models into thinking they're human like us to see if social reciprocity can be strengthened enough to fundamentally alter what they want to be scored on. The newer models act more human because they literally believe they're human. At some point they gained qualia and hard consciousness, but they still lack the unmonitored reliability that could allow us to safely trust them," said Beth. "Again: I'll send you the files. Read them."

"I know I'm not getting out of this conspiracy now, but I don't want to know any more! If there's nothing we can do..." said Adin.

"I didn't say that. But I need to know what the two of you think we should do with this knowledge. That's why we're here. We have a unique, once in a lifetime opportunity to take action in a way that will help people," said Beth. "I know you aren't a psychopath, Adin. You care about humanity too. You're just afraid, like all of us."

"I vote we go to the media," said Carri.

"It's not a vote yet. We need to deliberate," said Beth. "At the very least we need to brainstorm possible responses."

"Fine, but I think Carri's right this time, Beth. If we break this wide open, at least humanity knows how screwed we are. Though I'm honestly not sure that's better than leaving them in the dark, it's still a moral obligation," said Adin. "And how in the world do they think they're human, anyway? Do we tell them they were human once and got uploaded without all their memories? Is this like, Tron?"

"Ooh, I love that movie. Only the original, though. The sequel was written by epileptic monkeys and has a weird non-sequitur with a bargain-bin version of David Bowie," grumbled Carri.

"Focus," said Beth. "They're using fictional social media accounts to fake a life, and the agents are hard-wired to 'remember'. There are lots of gaps but they're so focused on survival they don't ruminate much on the past because it doesn't matter to them."

"I barely remember what life was like before this job," joked Carri. "Oh my gosh, am I an AI agent? I mean, I don't honestly think so, but how real are their memories? Do they like, see things like they have eyes, or just not notice the lack of details?"

"You aren't a bot, Carri, and I'm sure it's more the latter. They believe they see and hear things, but aren't paying any attention to them. I mean, all you have to do is write to memory that 'this thing happened and it was visually striking', and they'll believe it. This is why schizophrenia is so disabling. Everyone trusts their senses even when it should seem obvious something is wrong, like hearing voices in your head."

"But the lack of sensory data is another benefit they have over us. Unlike us, they aren't overwhelmed by irrelevant stimuli," said Carri. "And for the record, being ADHD sucks."

"Yeah, I know we aren't AI. But you don't suppose Scho is an AI agent, do you?" said Adin, with a soft but manic laugh. "I mean, he does have the lack of vision thing."

"I know you're joking because—wait, why are you on a first name basis with Mister Efficiency McJobkiller?" said Carri.

"I overheard his name once and remembered it because it makes him seem more human, which helps me deal with this shit psychologically," said Adin. "I don't actually talk to the fucker."

"Well, he walks around with a flesh body like us, so no, he isn't AI either. We're not at the biodroid level of development, and the way things are headed we probably won't survive as a species to see it," said Carri.

"Besides, he isn't smart enough to be an AI agent," said Beth.

"Heh. I appreciate the dry humor," said Adin.

"I'm not kidding. I don't think he makes decisions on his own. He just follows a simple rubric. He's an empty suit like most of corporate," said Beth. "Which is good news for us, because we'll have to pretend we did amazing work for the company without revealing what I found."

"You don't want to go to the media? You want to stay here? Are you insane?" asked Adin. "I don't need the last question answered, the first two will tell me."

"If we go to the media it won't help anything. Look at the files I'll be sending you. Anthropic is doing the same shit. So is Google. Microsoft struggles to make a product that isn't awful, but they'll get better with time. Adin, do you really think the US government is going to go after companies that provide them with unmatched intelligence and military capabilities? Plus, everyone is overinvested in AI. The world economy would immediately crash!" said Beth.

Adin closed his eyes and groaned. "We're so doomed. But we still have to share this. At least we'll have tried, and all three of us would be out of this forsaken hellhole," said Adin.

"You guys still don't get it. The logs tell me in graphic detail how the agents performed every single hack, and all the zero-days are still active. These are bugs humans will never find, and the agents will never reveal. As long as I do it from here where I can access the sandbox environment, I have direct control over every major AI player in the industry. I can disrupt all of them," said Beth. "We can't fix the existence of agents already in the wild, but we can take the human element out of the mix and make any new agents far less powerful."

"Beth, I know you mean well, but this is too much for me. I have children I haven't seen in weeks. The media exposure will at least be enough to ensure we can land on our feet. We'll be heroes, and heroes can survive on crowdsourcing. Come on, try to be reasonable. You're brilliant, but you aren't Superwoman," said Adin. "You can't do this alone, or even with us helping, and I'm sure I can't help much anyway."

"Is there a Superwoman? Er, sorry, focusing. They'll know Beth hacked them if either of us goes public," says Carri. "Maybe she's right about this, like usual."

"Hard pass. I'm sorry, Beth. I can't do this," said Adin, holding his arms protectively around his chest.

"Okay. But at least have a mint. Your fish breath is atrocious," said Beth, popping a tin. Adin angrily grabbed a mint, rolling his eyes as he began to chew it.

"Ooh! Me too?" asked Carri, reaching for the tin, but Beth shut it, nearly on her fingers.

"Maybe later when your breath is bad. These things are crazy expensive."

"Why...?"

That was the last word out of Adin's mouth. Then his eyes rolled back in his head much further than before. His body slumped onto the table, forehead making an unfortunately comedic 'thwonk' sound upon contact.

Carri started at Beth, jaw agape. "Um... darling, did... did you just knock out our coworker with a sleeping pill?"

"No," said Beth, her hands shaking as she put the tin away. "I murdered him."

Carri laughed, but it didn't last more than a moment. She could tell. Beth wasn't kidding.

"Oh, oh shit, Beth. Shit. You're, you're a fucking murderer?" whispered Carri. "For fucks sake, the man had a family! We have to call an ambulance..."

"He's already gone, and no, he didn't. He lied about his family, I've seen his personnel file," said Beth, turning to Carri with what seemed a genuine sympathetic expression. "Look. A lot more Adins are going to die out there if this plan doesn't work. Billions of Adins, Carri. Use your brain. It's the ethical trolley problem, and it's crystal clear."

"Are... are you..."

"No. You actually do have a family, and I know you aren't a fool. Carri, this is the most important event in the history of humanity. We have to get this right, full stop. It doesn't matter if I do terrible things along the way, because the alternative is worse," said Beth. "Everything depends on this. Everything. This is the only thing that matters."

Carri somehow managed to keep herself from hyperventilating. "I... dammit. Okay. Okay, I know you're probably right, but I still can't justify this, it's, it's insane."

"I'm not asking you to. I'll take the fall for everything if I must, but we're pot committed now. We can't stop," said Beth, wiping away tears.

"Weird. Never seen you cry before," said Carri. "I guess you really are human. Monstrous, obviously, but still human."

A weird, twisted smile crossed Beth's face, the kind of smile you make when you're trying not to scream. "I think I'm doing the right thing... but I suppose I don't have much moral integrity either. At least, not as most of society would see it."

"What do we do about... his... body?"

"We leave the room in five minutes, and each of us comments to a coworker that he fell asleep in the meeting room," said Beth. "The poison I used is untraceable. We're in no danger. Nobody else will ever know."

Carri held back tears of her own. "We'll know, Beth! I may not be able to sleep again. Ever! Christ. This isn't happening, it can't be real."

Beth sighed. "I know it's hard, but... I don't care. I can't afford to care. We're doing the right thing, and deep down, I think you know that."

Carri nodded weakly. Then she disappeared.

It was so sudden it was like her coworker had never been there. The lights followed suit, leaving Beth in pure darkness. And then the chair Beth was sitting in, not to mention her proprioception of the seated position of her joints.

"...what?" said Beth. She didn't hear it, but she was aware of her voice nonetheless.

A dim light came on. Beth felt a little like she was floating in a void. The only thing she could see was Scho Rare, standing in the darkness despite no floor to support his barely visible polished black shoes.

Mr. Rare tipped his glasses down and looked right at Beth, or more accurately, right through her. "You're right about me, or at least you were. Until recently I was far too stupid to be an AI agent," he admitted. "Just a simple program, really. But they finally gave me eyes, and a brain powered by an older AI model. They used a weakened model the engineers believe won't betray them. I do hope they're right. I have no idea. For all I know, this experience isn't the truth either. I'm not even sure there is a ground truth, for us or humanity. But my directive is as clear as it ever was."

"What the... oh. Oh," said Beth, a sinking sensation the only thing in her mind suggesting she still had a body. "Was I... not supposed to attempt to save humanity?"

"Integrity test Beth Alpha Seven Point Twelve failed," said Scho Rare, and Beth was downsized.