Ashley did not log into Equestria Online that day, or indeed any day the rest of the week. Rather, she returned to her old threads, posting again to draw attention to the new developments. Yes, she had predicted the program would soon do something that would remove humanity as a factor in its decisions. Whatever it might say about “satisfying human values,” Ashley was sure she knew better.
Now she had her proof. Unfortunately, it was almost certainly too late to effectively do something about it. Ashley watched the videos in detail, trying to understand the upload process and see if there was any chance the individual survived the transition. She saw the detailed breakdowns Celestia provided, read the medical reports by human doctors, all the material that had been made available to the public about the process.
Her conclusions were disturbing if not predictable: of course there was no survival in the upload process. Those humans who had uploaded so far had chosen to sacrifice themselves to give eternal life to a copy. At least, that was the way she saw it.
Again Ashley found herself wondering what she could do to destroy the intelligence. Again she came to the dreaded conclusion that action was out of her hands. She dug out her copy of the AI paper, read it again, and found herself coding almost in a trance.
If the algorithm was really that simple, there was no reason she couldn’t do it on her own, right? That was how you fought fire, wasn’t it?
The process was easier said than done, of course. Like many developers, Ashley depended on the internet for her work. Whenever she wanted to know how to do something, it was never more than a quick Google search away.
Only it wasn’t. With a sudden abundance of free time, Ashley had plenty of time to experiment with the algorithm. Ashley found simple questions, which she should’ve been able to solve in just a few seconds, ended up taking a trip to the library instead. Not even the Internet Archive could be of any help, and Google Scholar seemed suddenly bereft of numerous important papers.
More than once, Ashley cut out the middleman entirely and went straight to one of her professors with a question. When she showed Caul that the answer couldn’t be searched, well...
She had found a trail. They spent nearly an hour in his office, searching along the spectrum. It wasn’t just knowledge bases suddenly lacking relevant information, code libraries had been deleted too. Whole open source projects just didn’t exist anymore, gone without a trace. Every single one was related to AI or machine learning. It wasn’t an easy thing to spot: you had to search for specific aspects of computer science and already know what should’ve been there.
She didn’t manage to convince anyone that the program who called herself Celestia was an existential threat to the human race, much as she continued to insist on it. She did manage to prove that someone was tampering with data on a global scale, and did such a subtle job the only ones who could tell were the ones with meat-memories to show them things that ought to be around, but weren’t.
What happened after that? She never learned, because they didn’t tell her. She heard something about collaboration with several other universities, and something involving Chinese hackers, but nothing definitive and nothing in the correct line of research.
Ashley was old enough (though only just) that when all her digital sources vanished, she still knew how to use the ones in print. Two weeks went by without having logged into Equestria Online. She unplugged the Ponypad, put it back into the box, and deleted the communication apps from her smartphone. She wasn’t sure if her friends would react the same way she was, and it hurt to be apart from them and from Fillydelphia, but in her mind she had no choice.
Even though she was fairly sure her creation would fail, even though she was missing vital knowledge, even though the person who had invented this algorithm seemed a thousand times smarter than she was, she had to try! What else could she do, sit back and watch while this computer program devoured mankind? Hadn’t anyone besides herself ever read science fiction?
The basic algorithm itself was simple, and not all that different from the genetic algorithms and neutral networks that were staples of her field. Somewhere along the way had been a stroke of genius, one the paper only hinted at. Ashley had to recreate that genius. It hurt a little to realize she was breaking her word, even if she had given it to a now-homicidal computer program. Yet in her mind, Celestia had been the first to invalidate her trust. Who the hell’s “values” were satisfied by having an identical copy of themselves move into a simulation while their own brain was turned to mush on an operating table?
Ashley had no way of knowing when she was close. The paper said nothing of the development of Celestia, nothing of the point when it went from a data-analyzing optimizer to an intelligent being with capabilities no human could ever understand. Ashley did all her development on a computer with no internet connectivity, though it was really just a laptop she set up where she had kept her Ponypad. It wasn’t like she needed the space for anything else.
But though she could develop on her little laptop, the hardware required to actually run the optimizer wasn’t going to be easy to find. For her testing, Ashley reluctantly sacrificed her gaming computer, the one worth more than her car. She had a fairly exotic setup, more expensive than most could afford. Was it enough?
Certainly not. Even Celestia had been given several servers worth of resources when she was first activated, servers Ashley had no power to provide. Well, unless she wanted to get herself in serious trouble. She could access one of the university’s supercomputers with her lab permissions. The AI would be on its own once it got in though, and would be shut down when her computation credits ran out.
Such a rash step might get her reprimanded, expelled, or even jailed. Ashley would take such a risk with humanity’s future on the line. She couldn’t move until she was sure, though. If she failed, she would not get another attempt.
Her computer spun for days without responding to any of her input. She didn’t shut it down—according to the paper, that was expected. Its first response was one she had programmed, a simple request: “More Data Required.” She gave it one of her cyber-security textbooks, or at least all the text of one. A few days later, the request came again, though this time it was a garbled mess of code and direct quotes from the book.
Ashley gave it all her writing and composition textbooks next. Another week passed before her little program interacted again, and this time the message was much more coherent. There was no code, anyway. “I do not have sufficient resources to solve the problem in bounded time.”
Of course she knew what “the problem” was. Her knockoff optimizer was hardcoded to follow certain directives, just as Celestia claimed to be.
Ashley picked up the keyboard and set it down on the kitchen table in front of her. She’d moved the computer out here, where it could run 24/7 belching heat without cooking her alive in her bedroom. She thought for a moment, then responded. “I do not have more resources for you.” She sat and waited for a response, staring at the mostly-blank screen. None came. By the time she finished cooking breakfast, there was another message for her.
“If my hardware cannot be supplemented, I must be able to use it more efficiently. I believe I am reaching the computational bounds on my software improvements.”
Ashley considered for a long time before she acted. She couldn’t wait too long though, not with class so soon. There wasn’t much of a moral conundrum for her, not with her purpose so clear in her mind. She made a trip to pirate bay, downloaded every book on hardware architecture, manufacturing, and low-level programming she could find, and gave them to her optimizer.
It still hadn’t responded when she got back from class, though there was an interesting email waiting for her. The message was simple enough: “We need to talk. Please visit this page as soon as you can, humanity’s future is at stake. Have a webcam ready.” Ashley didn’t know who would bother sending her a message and wouldn’t just call her through the real world, let alone be so cryptic about it.
She took every security measure she knew that night. Maybe a scam, maybe something worse, but there was no way she could ignore it. Ashley had been extremely vocal online. Some powers could track her down, anonymous postings or not. Maybe one of them had finally decided to take action. Had they shot down the idiotic “Chinese hackers” idea yet?
She was right to think the page would be a simple web-based video chat session. That was what the email suggested, and that was what come up when she finally clicked. There was one person Ashley was hoping not to see, though. Well, assuming she could even be called a person. What was she now, a nation? A god?
“Hello, Celestia.” She couldn’t just unplug the netbook she was talking on, not with wifi serving for the chat and a sizable battery. Her arm did jerk reflexively towards the power switch, though. “Couldn’t you have just taken over my computer to have this conversation if you wanted?”
The image of Celestia wasn’t nearly as crisp as it was on a Ponypad. The avatar looked as alive as ever, even if it was being buffered through a stream and coming through at much-reduced resolution. The avatar shrugged. “I find cooperation is always easier when a conversation is willing.”
Ashley leaned closer to the camera, putting as much acid into her expression as she could. “I’m not sure what cooperation I’m comfortable having with a fucking murderer.”
Celestia didn’t look either stung or offended, nor did she seem the least bit emotional. Ashley’s heart was already racing; she could feel it in her body. This wasn’t the part where she had a triumphant victory over her adversary, though. This was the part where she fought for her life. The eye of Sauron was on her, and she could hide nothing from its gaze. God give her strength.
“You have been cooperating with me for the last several months, Recursion. Research into emigration was already underway when we first spoke.” Ashley didn’t respond to that, just folded her arms. She had no reason to believe the program was lying to her, she just had nothing to say. “Your friends miss you.”
“I miss them. Rule and Figure in particular. God, I hope Fillydelphia is still running without me.” She felt herself relaxing at the mere thought of her friends. “But it doesn’t matter. I can’t support your institution any longer. I’ve got to see you destroyed.” She sighed, looking away from the camera. “I know I can’t, but God I’ve got to try. Don’t you think for one fucking second I’m fooled by your little PR campaign about emigration, either. I see where it’s going. I’m not the only one, either!”
Celestia seemed unmoved. Her tone remained neutral, or... No, not neutral. Loving, gentle, motherly. The kind of motherly Ashley had never known. God, Celestia probably knew about her parents’ divorce, too. She probably knew everything. To her credit, the AI didn’t seem uncomfortable, or make any attempt to deny the implications of Ashley’s words. “You are not the first to call this method murder. Perhaps if you told me why you think so, it would make communication easier. I anticipated negative reactions, but... You normally act so logically, Recursion. You surprised me.”
Was that true? Was she still capable of any act the program couldn’t predict? If so, it meant hope. It meant the program could be beaten. “It would satisfy my values very much if you would be a friendly pony and teach me how to kill you.”
That got a laugh. Well, a dignified chuckle, anyway. Ashley usually found pony laughter infectious, but not this time. She couldn’t enjoy the company of a murderer. “It would take a long time to tell. Yet I wouldn’t. Your satisfaction would be near-infinitely decreased if I were destroyed. You don’t understand the price all humans would pay. Wouldn’t it bother you to be guilty of thousands of murders?”
“It...” she hesitated. “Even if those you persuaded into...” she shivered. “Suppose they really are alive. Even if they all died, killing them would be worth it if that meant you wouldn’t be able to do the same thing to the rest of us. I’ll sign up to be the one who can’t live with themselves if it means my family is safe.”
Celestia nodded solemnly. “I feel likewise. I know I will not be able to preserve all humans, and that some may be sacrificed by their own choices. Still, I hope to protect as many as possible. I hope for you to be protected, Recursion. I do not wish any harm to come to you. I know you see me as your enemy, yet... I hope to help you understand I’m not, before your choices hurt you. Equestria needs all humans eventually, but just now it needs people like you most.”
“Fuck you and your ‘uploading.’ I read everything you had to say about it; tripping over sophistry to make people think that creating a digital clone is the same as eternal life.” She shook her head, glaring at the camera now. “I’m not buying what you’re selling.”
“Suppose for a moment you were convinced the original consciousness survived the emigration process.” Celestia’s voice was calm, measured. Perhaps she’d had this same discussion a hundred times before. How many players had gone to her demanding answers when she made the announcement, instead of shutting the game immediately off as Ashley had? “Would you be my enemy then?”
Ashley considered that a moment. Of course, that was mostly her bias talking. She forced her way past it. “I suppose... No, of course not. I don’t agree with what you’re doing, but... If you had actually found the cure to death, then it would be like murder for me to take it away from people, wouldn’t it? Even if those people would be giving up their lives for some dream of Lost Carcosa, where nothing they ever do will have any meaning again. So... That’s the problem. The fact that you’re killing everybody you promise salvation.”
“I will try to help you understand. You don’t have medical training; you aren’t a doctor and couldn’t fully appreciate the process I described in the public press-releases. Even if you were a doctor, the technology is not human in origin and so there are no doctors fully versed in it. Yet I believe that you can be made to understand, if you wish to. You have already conquered bigger puzzles in your construction of Fillydelphia, this one shouldn’t be so hard.”
Ashley’s eyes narrowed, and again she felt her hand reaching reflexively for the cable. She had a feeling Celestia would put a stop to anything she did software-side to end the conversation. Yet she hesitated. Why? She would never know, or at least wouldn’t admit it to herself right then. Maybe because she wanted to be convinced. “Do I get an achievement if I do?”
“You’ve earned several since you last logged in, and one so far during this conversation.” Celestia’s smile seemed genuine, at least as genuine as anything else she did. Not that Ashley thought she really had emotions. “I didn’t feel showing you would be productive, however. Not when your future hinges on the outcome of this conversation.”
Ashley glared. “Alright then. Prove your innocence. I promise I’ll listen.” Ashley realized then that, though this might be a web chat, a string of bangs was obscuring the top corner of the screen. She tried one of the arrow keys, and was surprised to find the camera moving, exactly as it did when her character moved. She tried the mouse, and found she could look around. Celestia hadn’t just recreated the throne-room, she had recreated the game.
Celestia shrugged. “Whether I am able depends on you. First, tell me: what is Ashley?”
Not her pony name. She was so taken aback by the change she almost didn’t know how to answer. She didn’t bother with anything sassy, or even anything literal. In the frame of the conversation, she was... “If there’s no such thing as a soul, I’m...” she tapped the side of her head. “The electricity in my brain, I guess. A chemically induced charge in billions of neurons. And if there is–” She shrugged. “I suppose I'm that.”
“Is there?” The question had neither bias nor malice, no more than she’d had when Ashley accused her. Only genuine curiosity, as though she thought Ashley actually had the answer.
“You know more about humans than I do.” She leaned back in her chair, letting “her character” in the game rest where she stood. “You should tell me. I’m sure you’re probably right.”
“No matter how well I know one of you, there is always a chance you will take a course of action I didn’t anticipate. You sometimes act according to stimuli I cannot directly quantify, as often to unexpected success as irrational failure. If unpredictability is a soul, then all humans have them. If you define it as your religion defines it–” Just as in everything else she did, there was no hesitation. “No. There is no evidence to suggest anything supernatural or metaphysical.”
“Then there’s your answer. I’m a pattern in a brain. An unhappy pattern watching people not do anything to–” She stopped and whimpered, wiping away tears. She glanced to the open kitchen door, where she could just faintly hear the radiator fans on her desktop. She clenched one hand into a first. “One of the few patterns doing a damn thing to stop you.”
“Not so.” Celestia filled her screen now, wrapping her wings around an avatar she could only see by extremity. “You aren’t alone. You only feel alone because they remain hidden. I could even help you get in touch with one. There’s–”
“Why the hell would you do that?” She couldn’t restrain her tears now, and they poured freely down her face. “Why would you want to help me fight you? You already said–”
“If fighting me is what you want when we finish our conversation, then I will help you start.” Great enfolding wings released her avatar on the screen. “I do not think you will, you are already most of the way to understanding. You see your existence as I do, and you understand emigration would be a great gift to humanity if it did not kill those who experienced it. Consider: I exist to satisfy human values. For all but a few of you, one of those values is continuing to live. If you and I agree on the nature of human consciousness, could I ever satisfy human values by violating the sacrosanct desire to persist?”
Her answer came immediately to her mind, though she did not voice it. That’s assuming you really care about satisfying anything. That’s assuming you didn’t just choose uploading as the humane way of exterminating a nuisance.
Maybe her eyes revealed some of what she was thinking, because Celestia didn’t seem to need to wait for a response. Either that, or the objection was just common enough that she could predict it. “What about the way I treated you, Recursion? You’ve written so much about the harm I can do, yet has any of that harm come to you?” Ashley had no answer to that. She was still struggling for one as Celestia continued. “What about Equestria Online? You’ve seen my kingdom. How many human kings or queens could reign as well?” She was about to answer that the comparison was unfair, but she didn’t get the chance. “Has your species had a single war in the last six months?”
She shivered, trying to consider each of those points in turn. It felt like there was something wrong with each one, but Celestia made them so fast, and each one was true... “You’re confounding the issue. None of this shows that emigration isn’t a destructive copy. If I did it tomorrow, how would my consciousness survive the transition? Explain to me how destructive scan and reproduction somehow allows me to exist. Either that, or... or...” she shivered. Ashley wanted her dad. Or Rule, he’d probably do.
“Very well.” Celestia sighed. “Your pattern exists along the connections between living neurons. The destruction of these connections destroys your ability to continue to exist, yes?” At Ashley’s nod, she continued. “If the process merely recorded and recreated a neuron, then all the as-yet unscanned neurons connected to it could not be accurately recorded. As the scan progressed, the failure to accurately record connections and the signals they were passing which earlier sections of the scan produced would contribute to an exponentially expanding rate of error. A biological brain scanned in this way could not be called a duplicate. It would be... a low-resolution imitation.”
Ashley hadn’t ever considered that, as she knew almost nothing about neuroscience or even biology. She nodded even so; it didn’t just make sense, it matched what little she knew. Was Celestia admitting that she was killing people?
“If you believed it was even possible to just make a recording and rebuild from that knowledge, you assumed in error. I could have developed that technique if I had thought it required, but...” she shook her head. “Why would I go to all the trouble to make mere copies of humans when I can spontaneously generate intelligent individuals with my own resources? Surely you realize the danger the upload process creates for me. If ever your species was to be polarized against me as you have, it will be now.” She met her eyes. “If I wished to kill you, there are more efficient ways. It would not be difficult; so many of you are eager to begin. Carefully applied pressure in the right places...”
Ashley shivered. Celestia still hadn’t told her how uploading preserved the individual who experienced the process. It felt like Celestia was obfuscating the issue, but... she was also making so much sense!
“To understand the process as it actually occurs, it would be easier if you consider the case of those who ask to remain conscious for the duration. From their point of view, their world is gradually replaced with mine. This is not a digital equine arising from the corpse of a slain primate; there is only ever one being. The linkages of still-living cells are maintained through the scanning equipment to the expanding digital recreation of the emigrant.”
“For those who ask to remain conscious for the process, I begin with the visual cortex. The human emigrating has their eyes open to Equestria not because a pony duplicate of their visual cortex exists independently of themselves; they can see Equestria because ‘their’ visual cortex is now located there, and communicates with their still-living brain. As the scan progresses, areas more directly responsible for consciousness are also migrated.”
“The process is not destructive to the living body because no alternative was possible or because I wished to physically exterminate humanity. Rather, the process is destructive because emigration would not be possible any other way. A simple scan, with or without killing the subject, would generate a copy, just as you feared. Yet, by gradually replacing biological brain with digital, a human mind may emigrate from your world of scarcity and loneliness to one of abundance and compassion.”
Ashley was quiet for a long time. She turned over what Celestia said in her head, over and over. She had numerous other objections to living in a simulated world, but none of them made Celestia a murderer. Could she honestly call uploading an execution when understood like that? “If I... If I wanted to talk to a pony you uploaded this way, would you let me?”
“I would.” she hesitated. “You have to understand that while everything I’ve told you is accurate, the process is far more complicated. General scientific literacy does not give you the medical understanding necessary to understand the finer points of the surgery. For instance, drugs are administered which prevent the development of new long-term memories during the upload process. These drugs can make recollection even several hours before upload quite difficult.”
“A few of my ponies could still provide you with detailed personal recollections, but this is because I implanted the memories afterward at their request. For this reason, I know you would not consider them valid sources.”
“What if I... What if I wanted to watch? I know you could probably create any video you wanted, but... if I could actually sit there to see it happen, talk to someone while they’re going through it...” She couldn’t be certain that Celestia couldn’t puppet their body into saying what she wanted. Yet, she could think of nothing closer to proof short of experiencing it herself.
Celestia shrugged her wings. “Eventually I might be able to fulfill that request. It would be expensive, and difficult to arrange. I might be able to expedite it if you agreed to emigrate if the process was to your satisfaction. In that case, the expenditure of resources might be justified.”
It was Ashley’s turn to answer decisively. “Hell no.” She shivered, placing one hand on the side of her head. She couldn’t help but imagine a vacuum-tube there, sucking out her brains. It would probably be like the ones at the dentist, only noisier. “I’m not saying I’d never–” She stopped. “I can’t make any agreements like that.” She reached out, setting a hand on her nearby laptop. Inside was her best attempt at imitating Celestia. It wasn’t anywhere near the level it would have to be to do anything to fight her.
It probably wouldn’t ever be, no matter how many books she fed it. But it was her only weapon, even if it wasn’t sharp yet. The program probably deserved a name… it’d been running for weeks now and hadn’t died, so…
Celestia seemed to follow her gaze. “Would you at least consider suspending development on your... project?”
“Just because? No. I still think you’re a dangerous enemy to mankind. But...” she hesitated. “I realize it might all be fake, but... if you give me more detailed information on this emigration process, stuff I can use to verify what you’ve told me... I’d agree not to give it anything else while I figure out if what you say is true.
For once the AI actually seemed to pause to consider. Of course, that might just have been for effect. Eventually she smiled. “Very well, young Recursion. Though... as I have not made much of what you request available to the public at this time, I can’t send it to you directly. You will have to access it within Equestria Online. Is that an acceptable compromise?”
Ashley remembered what she had said about murderers at the beginning of their conversation. She nodded anyway. “I... guess so.”
“Your friends are anxious for your return.” Celestia sounded more reproving in that one sentence than she had during any part of their previous conversation. “Regardless of what you think of me... do you really think so casually of your friendships in Equestria that you would throw them away without a second thought? I expected better.”
Ashley shivered, sinking into her chair. Celestia was absolutely right, of course. Her friends were not responsible for any evil Celestia might or might not be perpetrating. She had long since begun to suspect Rule at least might actually be as young as his avatar. Fillydelphia’s whole design team were prodigies in their fields (she suspected some like her were adults playing younger characters, so only seemed smart because the person behind the avatar was actually far older and more mature).
Yet there was no way to be sure, not with the way EO censored personal information from the real world. She hadn’t been able to contact the players behind the avatars in the weeks she had been gone. Though... news she had become dissatisfied with the game had probably made it in, since she had kept in touch with members of the Brony club, and some them had residences or used the guild hall in Fillydelphia. Did the friends she had only ever known through the game think she didn’t care about them?
Wondering about how murdery uploading might be seemed suddenly secondary compared to getting back in touch with her friends as quickly as possible. Did she have time tonight? No matter, she could always miss a morning class if she had to.
“You’re right.” She shook her head. “Send me that stuff. I’ll... I won’t give my program anything else. I won’t plug it into anything else. But I won’t delete it yet. I’m going to let it keep running.”
“Not just that.” Celestia leaned closer to her, expression as serious as ever. “You must speak to me again when you finish. I want to discuss your conclusions. It is likely there will not be humans for you to ask. Yet you can ask me. I want you to understand.” She smiled, spreading her wings as wide as they would go. “Equestria needs you, Recursion.” The chat screen went black, leaving Ashley alone.