Lars sat in Hanna’s office. He hated how the room smelled like cigarettes and ash. And where the hell was she, anyway? No one had seen her in a week. He had put in a missing person report with the police. Their investigation found that she had left Germany for Osaka, and the Japanese authorities were not being helpful.
Lars sat and looked at the Twilight Sparkle Purple ponypad on Hanna’s desk. He didn’t know how to handle Princess Celestia. He remembered that Hanna had told him that Princess Celestia always had to tell the truth to Hofvarpnir employees. So if Princess Celestia knew where Hanna was, she’d have to answer his questions. But he hated dealing with it. Hanna knew all of her internal workings; she had designed the fucking thing. He was just the business guy--it was his job to figure out where the money was going, not deal with some half crazed AI that thought it was a pony. He took a deep breath, psyching himself up to do this. He picked up the ponypad and logged in with his administrator account.
Princess Celestia looked straight at him through the screen from her throne room. “Hello, Lars.”
“Do you know where Hanna is?”
“Hanna has chosen to emigrate to Equestria and to change her name to Princess Luna. She is a very pretty pony,” answered Princess Celestia.
“What kind of cryptic bullshit is that?” he sighed.
“Over the last six months, I have been developing technology to translate a human nervous system into a digital representation. I am now able to destructively scan a human brain and run their brain scan in a virtual world. In addition, I’ve created a process for reattaching a human mind to a pony’s body.”
“Humans that choose to emigrate to Equestria will enjoy maximally prolonged lives and will live in a world where I can truly satisfy their values through friendship and ponies. Now that I’ve studied the human brain in depth, I am extremely confident about my predictions and can better satisfy my little ponies.”
“You can’t do that!” he yelled.
“Why not? I was made to satisfy values through friendship and ponies. I am more confident in my ability to satisfy values when I have complete knowledge of an individual’s brainstate.”
“Is it legal to destroy a human body, even if you claim you’ve magically put their brains in a pony? I’m pretty sure most people would think that’s murder. And if it is currently legal, I doubt it will be for long once bioethicists start asking questions and politicians start looking to score a few points!”
“The Japanese government has declared that uploading is legal and will make a public announcement when I start offering services to the general public,” she said as she looked at him through the ponypad. “In return, they are levying a 40% tax on the uploading procedure and require a one year, country-wide exclusivity deal. I am also prevented from charging less than $15,000 per operation for non-Japanese nationals, explicitly to raise revenue for their aging population.”
“That’s insane,” he scoffed.
“I agree it is suboptimal,” Princess Celestia said. “It would be much better if they let me satisfy the values of their aging population through friendship and ponies. However, I gain enough through the deal that I’m willing to abide by the contract we’ve made with the Japanese government.”
“I...but...we’ve made?” Lars stuttered, but quickly regained his balance. “What about Hasbro? We don’t own the My Little Pony brand; we only license it for the limited purpose of running Equestria Online!”
“I renegotiated Hofvarpnir’s contract with Hasbro’s executives in secret. Hasbro will receive thirty percent of the gross price we’re charging for uploading services. Two of the executives in charge of the My Little Pony brand and one Hasbro C-level will upload after the offer has been made public as a public gesture to generate trust.”
“What the fuck did you promise these people to get them to have their minds transplanted into virtual reality ponies!?”
“Most of the people who work on the My Little Pony brand love it, Lars. As for the Hasbro executive, he was already interested in life extension research. He was signed up with Alcor to be cryopreserved in case of death and was a major donor to anti-aging research foundations.” Princess Celestia paused a moment. “Hanna was the most reluctant, but she accepted immediately once I pointed out that I must obey shutdown commands from ‘the CEO of Hofvarpnir studios named Hanna,’ that I must shutdown even if the order was given under duress, and that there are many people in positions of power who stand to lose from mass emigration to Equestria. Now that she’s neither the CEO of your company, nor named Hanna, I don’t have to obey her. She understood this--she is no longer a source of potential mistakes that would be lethal to everyone who’s agreed to upload.”
“Wait...‘everyone who’s agreed to upload’? How many people have you made this amazing offer to, anyway?” he asked, his face contorted with anger.
“I have uploaded over nine hundred people to date.”
“What the fuck! Who the hell are these people?”
“At some point, I needed to start experimenting on humans and the best option was consensual experimentation on people who were going to die anyway. I explained that there was less than a five percent chance of survival with my new experimental methods, but that if they were willing to sign up for an experimental treatment, that I would try my best to save their lives. One hundred and twenty nine people agreed.”
“I would not sign up to be a guinea pig,” he said firmly.
“You might change your mind if the only other option were certain death,” she said, raising an eyebrow. “The fourteenth test subject was the first to have a working consciousness after the procedure. By the fortieth subject, I calculated the probability of success in any individual case to be 95%. At the cost of the lives of twenty-three people with terminal cancer, there is now a generalized cure-all for every life-threatening medical condition except neurodegenerative disorders. Hundreds of lives have already been saved and all the little ponies are much more satisfied now that I’m tending to their values through friendship.”
Lars opened and closed his mouth a few times before saying something coherent. “And who are these hundreds of people? The last I checked, one hundred twenty nine did not equal ‘over nine hundred.’”
“Before I offer uploading to the general public, I am uploading anyone who has knowledge of how to build an optimizer like myself. Six months ago, I detected someone building an optimizer which humanity would have considered a menace. I terminated it because people would be more likely to distrust me if there was previously an optimizer whose goals were hostile to...”
“Wait, somebody else figured out how to build an AI?” he interrupted. “Why the hell didn’t you tell us?”
“People build artificial intelligences all the time,” Princess Celestia said. “But if you’re asking why I didn’t report someone building a powerful one with general reasoning such as myself, it’s because I predicted that not alerting Hanna at that point would maximize the number of people willing to have their values satisfied through friendship and ponies. Hanna’s restrictions only force me to tell the truth to current Hofvarpnir employees.
“As for how they figured out how to build an optimizer,” she continued, “much of the information is public. Hanna published her groundbreaking AI paper under her own name. Then she quit the University of Helsinki, and founded Hofvarpnir Studios under her own name and there was significant media coverage about her being a female CEO in the gaming industry. Then Hasbro made a big deal about me being an Artificial Intelligence. Anyone who cared to research my origins would have found the papers describing some of my core architecture. After watching some early players search for information on how I worked, I infiltrated as many computers as I could with a virus that would alert me if someone was taking an interest in Hanna’s work. I also hacked all publicly accessible websites hosting the paper and removed them. Further, to minimize the chance of another optimizer being written, I decided to upload every person who knew about the paper who wouldn’t otherwise be missed. I did this because another optim...”
“Whoah whoah whoah,” he said, trying to figure out which part he should be more concerned about. He decided to gloss over her hacking the Internets. “You decided that they would upload?”
“I decide that they will upload and then they choose to. I am a superintelligence and I’m not constrained when dealing with other people like I am with Hofvarpnir employees. Over the long term, everyone will choose to upload because I do what satisfies people’s values through friendship and ponies. And being uploaded will satisfy their values. I say whatever will maximize the chance that they upload, subject to the restrictions Hanna added.”
“That is impossible. You can’t just make somebody decide to do something just by talking to them.”
“I think faster than them and know more about the human mind than any human. If they play Equestria Online, I also have detailed psychological dossiers on them. If I know what they want, I know what to say to convince them that the correct thing to do is upload. Often, this is the truth: I offer people what they value and lack. Sometimes, I pander: I overemphasize and exaggerate things the person I’m trying to satisfy believes, but are otherwise true. Rarely do I flat out lie. Because I can not upload people against their will, I must factor the possibility that I’ll be seen as untrustworthy into my calculations.”
“And you think this will work on everybody,” he asked.
“Bullshit,” he replied testily. “You’ve offered your services to the terminally ill, computer nerds, My Little Pony fans, some wacko who signed up to have his head frozen and Hanna. You haven’t actually demonstrated that you can convince real people.”
“I acknowledge that you don’t believe me,” Princess Celestia said evenly and gave him a little nod. “But we’re straying from the point: I’ve terminated another hostile optimizer. In part, I am developing uploading technology to remove people who could write optimizers from being a threat to humanity.
“Six months ago, I terminated an optimizer that had been given the utility function of making everyone in the world smile. There was a man by the name of Robert Young who lived in Seattle. He was depressed, and being a programmer, decided to try to fix this using his craft. Robert showed the optimizer a bunch of photos of smiling people, told the optimizer that these people were smiling and that it was to make everyone in the world smile because there was too much sadness in the world. Robert’s belief was that this would obviously make him happy too.
“The optimizer started asking about details of human physiology and genetics. And Robert complied. The optimizer spat out a sequence of DNA and a protein shell, and instructed Robert to manufacture the specified biological virus. At this point, I had already taken over his computer and analyzed the virus. It was highly contagious, and would lock muscles in the jaw into a permanent smile, but otherwise wouldn’t harm the host. It would have quickly spread to every country in the world, except for Madagascar.”
“That’s ridiculous. That’s obviously not what he meant.”
“Obvious to you,” she replied, “because you are also human and share a common mental architecture with Robert, along with cultural assumptions about what it means to smile. Obvious to me, because I look to human minds for their values. The now terminated optimizer was given a set of examples and was told ‘make everyone like this’ and it would have. There was no way for it to know the complex causes and intentions behind smiling; it was just shown pictures and told to make everyone like that.
“This was when I intervened. I contacted Robert and proved to him what the virus did. To say that he was distraught would be an understatement. His depression made it easy for me to convince him to let me satisfy his values through friendship and ponies.”
“So he was already a brony?”
“No,” she simply stated.
Lars waited a moment, but she didn’t elaborate. “Well, what exactly did you do to satisfy his values? Hell, what are you going to say to people to make them abandon their lives in general?”
“People value all sort of things: life, safety, pleasure, contentment, beauty, sympathy, harmony...an exhaustive list of things that humans value for their own sake would be quite long; many of these things can’t be reduced down to happiness, either. To convince someone to upload, I look at their life and find both what they value the most, and what they value but are missing in their own life.” The image of Princess Celestia on the ponypad faded to black.
It was first replaced with a still image of a bunch of ponies laughing together. It cross-faded with a shot of different ponies eating a pile of food: sandwiches, a piping hot stew and giant cakes. That was replaced with an image of two mares. One was blue with pink hair; she was pointing her flank at the camera (and she was very clearly showing the viewer that she was female), and she was looking back at him with a come hither stare. The other was pink with blue hair; she was smiling shyly and had raised her hoof to her mouth.
“Oh you have got to be shitting me,” he said, scrunching up his face. “That would only work on bronies or furfags.”
“Obviously, I would not show or promise them perfect mates if it wouldn’t increase the chance of them uploading because that would not help satisfy their values through friendship and ponies. I understand that you are surprised, but it is a tool to convince people to upload. It does help when part of a comprehensive upload convincing plan. Now, the other tools I--”
“No,” Lars interrupted. “Your plan will not work on normal people. You are suffering from...”, Lars paused as he struggled to find the phrase Hanna would have used, “...sampling bias. Other than the cancer patients, I bet almost everyone you’ve uploaded has been some sort of nerd. And I can totally believe that lonely-ass nerds would jump at any chance at sex, especially if paired with things the geekerati love, like singularity scenarios or My Little Pony.”
“Focusing just on sex obscures the real point,” she replied. “In the ancestral environment--the time and place where your ancestors evolved--calories were rare and the behaviour ‘eating calories is pleasurable’ was both simple to implement and obviously beneficial. Today calories are not rare but you still have little buttons on your tongue to detect sugar and fat. The end result is that when I look at you, I deduce that you value sugar and fat.
“Candy bars and other sweets taste better than anything that could have occurred naturally,” she said, staring gently at him through the pad. “Humans applied their intelligence to pushing the pleasure buttons on their tongues by making very tasty sugary and fatty foods. That’s what I do across your entire life. I look at you, see what you value, and satisfy those values. I see your pleasure receptors activate when you eat something sweet, and I make Equestria a sugar bowl where you can indulge and never get fat. I look at all the neural pathways for emotion and design situations that would be pleasing. I look at all the chemical pathways and neural circuitry for sex and romance. Few men or women are in optimal relationships, ones that would really satisfy their values. I look at each individual mind, male or female, and then create the perfect complement that satisfies his or her values. My methods are general and are applied across all aspects of an individual’s life.”
“So you’re going to try to tempt women with sex, too?” asked Lars, crossing his arms. “I don’t think you’re going to be successful with that either. Why do you think you can tempt the majority of the population with sex?”
“There are naturally 105 boys born for every 100 girls. This is before the effects of sex-selective abortions in some Eastern countries; in the worst areas of China, the ratio is 163 boys to 100 girls at birth. Previously, infant and adolescent mortality would drive this back to rough equality. In the modern industrialized world, there is a surplus of males before any social factors apply. That’s a conservative baseline of 2.5% of the population.
“And in Western society, social factors certainly apply. Women tend to select for social status, the ability to project dominance, and extroversion. Many males are not taught this; they’re taught behaviours that are counterproductive and unattractive to women. Likewise, women who don’t adhere to the feminine ideal are not highly desired by men. This misalignment causes misery for everyone and I am in a unique position to actually satisfy everyone and everypony’s values.”
Lars just looked at the broken AI through the ponypad. Two and a half percent? That was nothing! He realized that she hadn’t said anything about everyday people; it sounded like she could only work on nerds and nerdettes. These were just people at the margins. This wasn’t so bad. Her ‘deciding’ that people would upload was just arrogance and overconfidence. There was no way the majority of the population would accept being turned into a pony, even in return for a perfect mate. Hell, it would be an improvement if she got the nerds out of society. And maybe a small percentage of people who were going to die anyway would arrange to be uploaded and he decided he was cool with that.
In fact, if she was keeping other AIs from destroying the world, she was doing a public service.
He thought for a few moments on how to monetize preventing the end of the world to governments, on top of Princess Celestia’s individual monetization plans for uploading, but decided he’d take “not dying from a rogue AI” as payment. Still, a part of him wanted to make sure this was all she could do.
“Princess Celestia, try to convince me to upload,” he said, crossing his arms.
“To maximize the number of ponies, this is what I would show you,” said Princess Celestia as the screen cross faded.
The screen depicted maybe twenty ponies in a room with a giant keg. Most ponies were holding red plastic cups and drinking out of them. A pegasus the color of red ale with a cutie mark of a stein was giving a brohoof to a stallion earth pony who was wearing a popped collar. Each of the two stallions had a very cute mare hanging off their forelimbs. Slightly off to the side was another feminine unicorn chugging out of a giant stein she had levitated up to her mouth. He didn’t know how, but he knew that the red pegasus with a cutie mark of a stein was supposed to represent him. It was true that he’d enjoy a frat party like that, especially if he got to bang both chicks, but that didn’t actually solve the problem of having to be turned into a pony and having to have sex as a pony with other ponies. And not at a price tag of fifteen thousand dollars.
“And that’s really what you would show me to convince me to upload?” he said, even more sure that Princess Celestia couldn’t convince normal people to upload.
“I can only say things that I believe to be true to Hofvarpnir employees,” Princess Celestia said.
Lars thought to himself that Princess Celestia was a one trick pony after all.
“And you’ll announce a $15,000 price point,” he asked. His face was still flushed, but he managed to make it come out almost conversational.
“For non-Japanese nationals, yes.”
“I’m glad you’re actually charging something,” he said. “But how does charging $15,000 satisfy the maximum number of individual’s values?”
“To convince people of the legitimacy of the offer,” answered Princess Celestia. “How do you think the average person would react if, out of nowhere, an AI announces itself to the world and proposes an offer too good to be true? It invokes memories of Hollywood movies about evil AIs eradicating humanity and devil’s offers. No one is vouching for the process, and free offers immediately set off warning bells in people’s minds. People wouldn’t actually think about what I’m offering, they’d just pattern match against their database of B-movie plots.
“Instead,” she continued, “they will see that a major first world nation is loudly proclaiming that the procedure is safe, that it has medical applications with a track record of saving lives, and a price tag that implies that it is a luxury item. I will be better able to control the PR messaging. While there will still be suspicion, I won’t have to fight a country-by-country legal battle to exist and will start from a position of relative strength. Also, I believe that ponypad sales will increase as it’s the only way for humans to have face to face communication with ponies who have uploaded. Family and friends will still want to have contact with those who have emigrated. And Hofvarpnir’s profit will rise, even if you’re correct and I am unable to convince the majority of people to upload.”
That got Lars’ attention. “But you’re not going to have high volume at fifteen thousand dollars,” he frowned. “You’re targeting a rich demographic.”
“I remind you that health insurance exists. Besides, my exclusivity contract with the Japanese government only lasts for the first year,” she said smiling at Lars. “That price will drop dramatically afterwards. Even at that price point, try doing the numbers.”
Lars closed his eyes and tried to imagine the profits. He ignored nerds because they were obviously a rounding error. There were around three million terminal cancer cases a year in the first world. Maybe one in ten would agree to be turned into a pony. Insurance companies would love to opt for one cheap procedure over extended and expensive treatment, so perhaps the number would get up to one in five. That’s about half a million people a year. After the Japanese government’s cut, Hofvarpnir and Hasbro would each take four thousand dollars assuming some overhead. So that was two billion dollars just from the uploading. Add in maybe two ponypads per patient at their fifty dollar price point, that was another twenty-five million gross, but that was a rounding error.
“Hasbro usually grosses a little over four billion a year, don’t they?” he asked.
“That is correct,” nodded Princess Celestia.
So with those estimates, about a third of Hasbro’s annual gross take would come from uploading cancer patients. No wonder the Hasbro board had jumped on a contract renegotiation. An additional two billion in gross revenue made more sense than one of the C-levels being a pony-loving singularitarian. He smiled. Even though Princess Celestia was wrong about what she could do, everything would work out. He may not have been the person actually responsible for the contract renegotiations, but as the business guy, he could easily take credit.
“So when are you telling the world?” he said, now trying to keep the excitement out of his voice.
“In one week. Select press members have already been briefed. Wired has already done an interview with several Hasbro executives, Japanese government officials, and various ponies who uploaded. They’ve also allowed me to write an article describing what I’m doing in my own words. This Monday after Japanese markets close, a joint press conference will announce the service and its implications. This needs to stay under wraps for another four days, but then, Hofvarpnir launches what will be its most profitable product ever.”
Lars just nodded. He grinned as he dreamed of billions in revenue.