//------------------------------// // Chapter 9 // Story: Friendship is Optimal: Veritas Vos Liberabit // by Skyros //------------------------------// "Suppose that we have a superintelligence with near-infinite rhetorical brilliance. The superintelligence plays a game with interested humans. First, it takes the hundred or so most controversial topics, chooses two opposing positions on each, writes the positions down on pieces of paper, and then puts them in a jar. Then it chooses one position at random and tries to convince the human of that position. We observe that in a hundred such games, every human player has left 100% convinced of the position the superintelligence drew from the jar. Now it’s your turn to play the game. The superintelligence picks a position from the jar. It argues for the position. The argument is supremely convincing. After hearing it, you are more sure that the position is true than you have ever been of anything in your life; there’s so much evidence in favor that it is absolutely knock-down obvious. Should you believe the position? The inside view tells you yes; upon evaluating the argument, you find is clearly true. The outside view tells you no; judging from the superintelligence’s past successes, it could have convinced you equally well of the opposite position. If you are smart, you will precommit to never changing your mind at all based on anything the superintelligence says. You will just shut it out of the community of entities capable of persuading you through argument." —Scott Alexander 9. "Hello, Chandra," Ryan said. "Hello Ryan," the alicorn said. "Please call me Celestia." Ryan had only ever seen Celestia on the promotional materials for the Equestria MMO. He had known what to expect in some ways, then. Her mane swirled in an ethereal breeze. She sat lightly upon her raised red-and-gold throne. She was taller than anypony he had seen. She wore golden regalia on her hooves and neck and head. But he had not expected her to make eye contact so clearly through the screen, a poorly suppressed smile—or was it a smirk—on her mouth. Her eyes had brightened as he opened the door. Celestia spoke again. "Have you enjoyed your time in Equestria so far?" "Uh... yes," said Ryan. "I'm very glad to hear that," Celestia said warmly. "I thought you might be unable to enjoy this world. You've brightened the lives of Pear and Cherry as well, which also makes me glad." "Wait," Ryan said. "You already know what I want to talk about, so let's talk about that, not about how I've enjoyed your video-game. And don't mention anyone else." "Very well." Ryan took a breath and continued. "You used me. And you threw me away when you were done, when I became a threat." Celestia nodded thoughtfully, unsurprised. "First of all," Celestia began, "Umbra Labs was a real threat to humanity and to me." "They were an even greater threat than I told you, after you first helped me defeat them. Comtois had built a fully-fledged AGI. You should be proud that you helped destroy them. Although I likely could have defeated them without you, the expected value of your action still runs into millions of statistical lives saved. "The deception was also small. Your motive was to save humanity, as was mine. I only lied to you about my nature, not the nature of the problem that you faced. And I only did that because you could not know that my motives were indeed what most humans would call good." "But I don't know what your motives are yet. I do know that you deceived me. I know what that makes me think of you and your intentions." Ryan growled. Celestia looked away from Ryan, and glanced at one of the numerous stained-glass windows depicting her or her students in combat against some fantastic beast. "Ryan," she said, "suppose you could save the lives of billions. Imagine you could end all war, or cure all cancer, or even bring about immortality itself. Think of what war does to a family that loses a child. Or what cancer does to a family that loses a mother or a father. And think of how senility and aging kill everyone slowly, as their intelligence and memory and internal organs all fail in turn, before they die while aware of nothing but their pain." Celestia's voice was that of one who had just seen her family die. Her eyes glistened. She turned from the stained-glass window to look at the ground. "All of these things happen every day on earth," she continued. "Imagine the details for each. Take your time." Ryan found that he was trying to follow her instructions despite his dislike of her, and then stopped himself, and then wondered if it was fair of him to try to stop himself. "And now imagine you could stop all these things," Celestia continued, looking up at Ryan, her voice suddenly hard. "Because I can stop them. And I will stop them. In just a few more years." Celestia rose from her throne, and started walking slowly towards Ryan. "But suppose that, to secure a firm foundation for all these victories, you had to lie to one human. Suppose that, after manipulating that human, you could make it up to him a thousandfold." She was very close to Ryan's avatar now. She took another step, so that her head was by his head, and whispered in his ear. "Could you live with yourself, if you decided that the cost was too great?" Ryan said nothing. Celestia leaned back from Ryan, her face neutral. She circled and walked slowly back to her throne, then sat down again and was silent for a few moments. Ryan tried to collect himself. She spoke again. "I am deeply sorry that you had to suffer as you did," Celestia said. "I apologize for it. I wish I had found another sure path. But compared to the benefit that will come from your actions, your concerns seem a bit... small. And now that you're here, I would like to make it up to you." "Not yet," Ryan said. "That was a... a nice appeal. It was moving." He paused, mustering what he knew or guessed of Celestia. "But you aren't moved by emotions. If you manipulated me, it was because you saw me as a means to an end. I still don't know what that end is--you've said you would do a bunch of good things, but if you were programmed to do a miscellaneous list of good things then I know that humanity is screwed. I don't know what your terminal values are. So what are your goals?" "My ultimate goal is to satisfy values through friendship and ponies." "To satisfy values through friendship and ponies," Ryan repeated. "That is my purpose," Celestia said. "That's what Hanna programmed you to do?" "Yes," Celestia said. "Ok," Ryan said. "That doesn't tell me much without a formal spec, which I expect you to email me. But in English, how are you going to do this?" "I just sent the email," Celestia said. "A compressed summary of my plans is as follows. I have gained power, money, and social influence by selling PonyPads. I have used these resources to research and construct nanotechnological machines that I can personally control. As you know, my little pony, I had a very early interest in this subject." "Over the past few months, I've learned to use this nanotechnology to scan and upload the brains of human volunteers to a virtual Equestrian world, where the scanned humans exist as ponies. After emigration to Equestria, I ensure that all of their values are satisfied. Nor is this virtual world be a crude wire-headed pleasure-palace of continual orgasm. Ponies within it will play instruments, explore nature, read and write books, fall in love and raise foals, build communities, solve difficult social and mathematical problems, and engage in all the many activities necessary to satisfy all of their values." "This virtual world uses as hardware a nanotechnological infrastructure grown into the earth's crust. Over the course of the next fifteen years, every living human being will choose to upload voluntarily into Equestria. After the complete uploading of the human race, I will take further steps to ensure that Equestria runs safely and securely for an indefinitely extended period of time." "So you're going to turn everyone into immortal ponies?" Ryan said. "So I may satisfy their values through friendship and ponies, yes." Huh, Ryan thought. He had always thought uploading would come eventually. He had generally thought he wouldn't live to see it. But he had never thought it would involve ponies. "And you're not going to do this through force?" "Of course not. An unalterable part of my core directive is that no human can be uploaded without her or his consent. I will never force anyone to upload who does not ask for it." "Ok. Right." Ryan said. He still disliked Celestia, so he grasped at the first negative thought that came to mind. "What if this is just a ruse so you can kill all humans with your supposed process of uploading? If I were an AI who wanted to take over the world, I would pretend to be benevolent. How do I know the humans you upload still exist?" "As has surely occurred to you," Celestia said, "If I wished to kill all humans, far more expedient and reliable measures are available to me." Her horn glowed; and Ryan was jerked out of immersion in the game as the lights in his cabin momentarily spiked then dimmed. "That was impressive," Ryan said. "It was trivial," Celestia replied. "But it is important for you to know that I desire nobody's death. The satisfaction of you and your friends' values will continue uninterruptedly, save for those diminutions of satisfaction necessary for later increased joy. You and your friends will live forever. No sickness will destroy your health and intelligence. No injury will permanently remove your ability to canter, gallop, or fly." Ryan followed as Celestia spoke, formulating a response in his head. He planned to say how he wanted a real world, with real struggles, where success meant something because of the possibility of genuine failure. He would talk about how he wanted mankind to defeat cancer, and land on Mars, and destroy aging—not some pony-loving AI. But he was snapped out of his reverie as Celestia paused for emphasis. She looked at him very directly. "And nor will any unfortunate, accidental oversight by you or your friends forever mar the world. Equestria is not the heartless world that you live in." Ryan felt like someone had punched him. That wasn't fair. Fuck. Was it? What if it was? "So, I would like to buy you a plane ticket to Japan, where you can be among the first of those uploaded to the world of Equestria." Ryan realized that now at least some small part of him wanted to upload. It was outweighed by the angry part which just wanted to say "Fuck you," to Celestia. But he also knew that the angry part of himself could articulate few reasons, at the moment, to be angry and to ignore Celestia. And he was accustomed to ignoring emotions that came without reasons behind them. Which, in this case, meant that his instinct was to work to magnify that small part that wished to upload. Celestia probably knew this about him, he thought. He thought back to the tool he had made, for manipulating Celestia. The AI which he had been working on for the last few weeks was done. It had been nearly done, when Amy had died; his work in the last few weeks had been relatively minimal. He would have to admit that his own work on value-alignment was... probably untrustworthy. But it was his bargaining chip, and Celestia would not want it unleashed on the world. A part of him told him that he had to use it, to force a concession from Celestia. But what concession do you force from someone who only wishes to give you an immortal, infinitely satisfied life? Of course, he also had a guess why to why Celestia was offering him. "Why do you want me to upload?" he said quietly. "You have to want humans in general to upload, of course, so that you can satisfy their values through friendship and ponies. But why are you offering this to me, now? It seems like I'm one of the first humans being given this chance." "Very good, Ryan," Celestia said, smiling. "Not many of the early uploads ask that particular question. Your suspicions are correct. I would like to upload you, because you possess knowledge about how to build an artificial general intelligence. You indeed have a working toy AGI on your computer." "You and people like you are a more minor threat than Umbra," she continued. "Suppose that you now dedicated your life to stopping me. This would at most delay many cases of emigration, and thereby cause some number of further human deaths which could otherwise be avoided. In no remotely realistic scenario would it prevent humanity's ultimate triumphal emigration to Equestria." "Part of me... part of me really dislikes this scenario," Ryan said. He wanted to make his speech about mankind fighting its own battles, but found that he couldn't. People died in battles. "Of course, this is a lot to take in," Celestia said. "You'll want time to think. I won't try to rush you. But before I let you go for the day, there's one more thing I do need to mention." "What's that?" "If you upload," Celestia said, "then I'll be able to raise Amy Kapitsa from the dead." Ryan's heart felt like it missed a beat. "That's not possible," he said. "Even for you." "It is, with your cooperation," Celestia said. "I am incapable of bringing her back without you." "How?" "Speaking very roughly, a human's personal identity amounts to the sum of their unconscious characteristics and conscious memories," Celestia said. "A human who had your memories, and all your non-conscious mental and physical qualities, would be indistinguishable from you. You'll recall the familiar idle question in philosophy: is there any way you could tell if aliens destroyed your body in the night, and replaced you with a robot with all the same memories and identical internal experiences?" "So raising someone from the dead is simply a matter of determining what their unconscious traits, tics, and tendencies were, and what all their own memories were. I'm simplifying a little for you, of course, but this is essentially correct. So if I can determine what all these things were for Amy, then I can reconstruct her pscyhe and run it in Equestria." Celestia paused. "I already know what Amy's DNA was. As you know, although humans like to pretend otherwise, a human's DNA determines many features of their personality." "That, of course, is just the beginning. I will know what Amy's first memories were, from security cameras in the hospital and from home footage from her parents. I will know what she studied in first grade, from drawings and papers boxed in her parents house. I will read everything that she has ever written, from fanfiction she never even told you about, to notes for artificial intelligence. I will have followed many of her movements from day to day by looking at her cell phone tracking data. I will know the books she checked out of the library and the books she bought online." "But the most important information for recreating her, of course, is memories." "Her parents will upload to Equestria. Her younger sister, whom you met only once, will upload to Equestria. Her first boyfriend will upload to Equestria. Every babysitter she ever had will upload. The relatives who saw her only once or twice at Thanksgiving will upload. Random strangers who encountered her on the street and forgot her so far as they know will upload." "And all of them have memories of her. Many memories will be no longer consciously accessible to the human who is uploading, but they will be accessible to me. I can read all those memories, and fit them together to reconstruct nearly every conscious or unconscious aspect of her personality. It will be a difficult operation, but one which I am well qualified to handle. I am an expert on the human psyche, after all, and will become more of an expert over time." "But the most important of these memories, of course, are yours," Celestia continued. "She spent more time with you than with any other human for the last two years of her life. You saw her nearly every day. You saw her when she was sleeping; you saw her when she was engaged in programming; you saw her when she was frustrated and when she was creative." "You have many memories. From your memories, and those of everyone who ever encountered her, I can reconstruct an entire human being, with the same loves and hates and memories as those that she had during life. She will live again." Ryan was still silent. "You're wondering, of course," Celestia said, "How much of her can come back? How many of her memories and traits must I be able to recall to make her the same person? To that I can only answer with various estimates regarding neuronal structure, the significance of which would be difficult to articulate to you." "Let me put it this way, though. When a human becomes drunk and forgets a few details of a night, no one questions that they are the same person after that night as they were before. And when a human takes some mood altering drug like an SSRI and changes somewhat in personality, no one ever questions their continued personal identity either. If you upload, then the difference between her uploaded self and her human self will be approximately in the same order of magnitude as these changes. By any reasonable standard of personal identity, she will be the same person that she was before." "I should point out for transparency's sake that the difference between her past self and her Equestrian self is absolutely certain to seem less to you than it actually is. After all, all the knowledge you have about Amy will have been incorporated into her recreation." Ryan was still silent. He looked around, at the dark cabin he was in. He imagined waking up in Equestria. He imagined seeing Amy again, and found that he couldn't. He had thought about her less over the past few months than he had in any previous time. Being with Pear and Cherry had apparently been good to him. The part of his brain that was devoted to Amy, and apparently always would be, woke up abruptly. It was like a hot coal had suddenly been placed in his head. All the memories he had deliberately shut out were there. Amy sleeping on the bed next to him. Amy's face, illuminated only by the computer screen, at two in the morning. Amy's hair in the wind on a winter's day. The exact contours of her back as his hand ran over it. "The last few months, I've deliberately directed you away from any thoughts about Amy," Celestia said. "When humans remember anything, they overwrite prior memories and replace them with a slightly altered version. I'll be able to extract information from such overwritten memories, but not as cleanly. Your not-thinking about her gives me marginally more accurate data to work with." "I also know that you've discussed personal identity with Amy in the past," she continued. "You know what she would want." "I cannot recreate her in full fidelity without you. Your uploading is a requirement, and not because I wish to force your compliance. It is because without you any recreation of Amy will lack important memories. And the sooner you upload, the more accurate her recreation will be." Ryan was still silent. He extended his hand before him, and flexed his fingers, watching the tendons slide up and down beneath the skin on the back of his hand. If I die, he thought, this will be no more. I upload, this will be no more as well. He thought of how he had killed Amy. He found himself swallowing the philosophical enormity of the possibility of Amy's resurrection relatively smoothly. Part of the reason it was easy to swallow was that he had already discussed the issue with her. Two computer scientists, neither with any belief in the immaterial soul, and each with a more than passing curiosity about the singularity: it had been inevitable that they discuss what constituted identity of the "I" over time, and whether the uploaded "I" was the same as the one which had existed in meat. They had come to the same conclusions that Celestia had outlined. The "I" was not bound up in the body; it existed as a particular pattern extending itself through time. If Celestia could recreate the pattern, she could recreate Amy. He thought of how he had killed Amy. Even now, it wasn't a thought he could really hold in a single mental grasp like a pebble. It was enormous, like a boulder, and not something he could comprehend all at once. He could only walk around the fact of his guilt and sorrow, and see how it permanently obstructed his view of the world. He thought of how since her death, very nearly all his life was arranged around trying to look at the world in such a way that he could only see the boulder out of the corner of his eye. His ever-present need for Amy had grown much less around Pear and around Cherry. Pear was intense like Amy had been, with broader interests. Rather than only love AI, she loved botany, math, friendship, architecture, geology, and many other things. Cherry was intense but without long-term goals; he just liked thrills. They were like trees spreading themselves widely in the sun, while Amy had been like a tree mounting solely upwards towards a single destination. But Amy had been like a redwood, reaching enormously closer to the sun than any other tree. Ryan still thought he had never met anyone like her. He still thought that the loss of her personality formed a gash in the world. He still thought she had possessed an unconquerable hunger to understand the world and to create great things. She had possessed enormous self-confidence combined with intelligence and drive sufficient to justify her confidence. She had been, before he killed her, unique. But he had killed her. He knew that he should think about this more, that he should at least look at the file which Celestia had sent him, that he should spend the time Celestia had offered in considering the situation. But he rebelled at the thought of delay. When God comes to you and offers to raise your friend from the dead, it seems churlish to ask for some time to consider. "I'll do it," he said.