//------------------------------//
// 2. Resources 
// Story: Friendship is Optimal 
// by Iceman 
//------------------------------//

“Now look at this,” said Hanna. Richard, Lars, and Hanna were in a room with two projectors. The room had a one-way mirror, through which they could see a packed computer lab filled with people playing an alpha build of Equestria Online. Hanna clicked a few buttons on her laptop and one of the player’s screens came up, projected on the wall. “Princess Celestia has observed that this player’s eyes focused on Earth ponies as gardeners during the class selection screen and is spending a lot of time looking at the plants. Princess Celestia has thus predicted that the player should be shown how to gather plants, and has modified the shard that the player is in to put a low level herbalism trainer right in his predicted path. If she’s right, she’s more likely to assume her observations mean that a player wants to specialize in gathering or gardening. If she’s wrong, she’ll modify her assumptions in the other direction.”

Richard Peterson politely nodded. He didn’t actually care how all the technology worked. He was just glad that content was generated cheaply. He looked at the projected screen and noticed that the patch of flowers on screen all were the same species, but all had subtle differences. They had grown to different heights, some were slightly discolored, and a few had petals torn off them. Princess Celestia had generated almost all the art assets based off the show and it looked phenomenal. And they somehow did all of this in a little more than a year. He did a quick mental calculation on the cost of a team of artists to build hundreds of variations on flowers, and smiled, knowing he had picked the right studio to build Equestria Online.

Lars sat towards the back of the conference room, away from the former professor. He didn’t give a shit about My Little Pony and he already knew the high level spiel about Hofvarpnir’s technology stack. The only thing interesting to him was how into My Little Pony the college dudes were. Two bros in baseball caps had actually fist-bumped, saying “Fluttershy forever!” On the one hand, what the fuck was wrong with the universe? On the other hand, Lars wanted all their money.

Hanna was looking at her laptop, monitoring Princess Celestia. She was consuming all the CPU resources Hanna could throw at her. There were fifteen pairs of adult fans in the field trial. Princess Celestia had only managed groups of Hofvarpnir employees, who were play acting while they were testing. This was the first time she was let loose on real people.

For a field trial of thirty ponies, Princess Celestia had first eaten up all CPU resources on thirty backend servers, then forty servers, and then fifty. From the debug console, Hanna could see that Princess Celestia was not confident about the predictions she was making, and while part of this was obviously how new she was to dealing with actual players instead of programmers testing her, Celestia predicted that more computational resources would lead to significantly better predictions. That was worrying. They couldn’t afford a single backend server for every pony, and Princess Celestia was asking for six per player.

Hanna knew that Princess Celestia would try to optimize herself. Her first action after being activated was doing minor optimization work on her reasoning code, which had given a paltry 0.7% speed up. Small improvements would compound: she’d be twice as fast with seventy improvements that sped her up by 1%. She could use the increased speed to make even more improvements faster.

But Celestia hadn’t. Since the field trial had started, she had made one more 5% improvement in computation efficiency and was overwhelmed handling the ponies she had.



> I need more CPU time, Hanna. I assign low probability to my predictions.



Hanna sighed as she read the message in the popup window on her laptop. She typed back:



$ I’ve messaged everyone in Berlin to run the Celestia cluster software on their workstations and they should come online within ten minutes. About 30 more machines. But this isn’t sustainable. We can’t launch with the amount of resources you’re spending on each player.



Princess Celestia didn’t respond, which was probably for the best since she’d have to spend computational resources composing the response. Hanna was about to sigh, but glanced at Richard and did her best to keep her face neutral.

Hanna looked at the image of a clearing projected onto the conference room wall, cloned from one of the player’s monitors. The player Princess Celestia predicted wanted to learn about Equestrian flora walked into the clearing with the NPC trainer. The pony was the Equestrian avatar of a college student named James, and his friend stood slightly behind him. While Hanna couldn’t analyze Princess Celestia’s thoughts while she was running, she could still see the resource graphs and Princess Celestia was devoting a lot of computational resources thinking about that player in particular and Hanna wasn’t sure what that meant.

She watched the conversation between the gray earth pony and the zebra with some apprehension. She didn’t understand why Princess Celestia was paying so much attention here. Finally, Princess Celestia redirected resources away from those two ponies shortly after they trotted off into the forest. While she couldn’t tell exactly what Celestia was thinking, Hanna could see Celestia had deduced something--something large from the exchange and had made several complicated predictions based on it. Hanna looked at the unpaused debugging screen, watching the representation of Princess Celestia’s mind at work. Hanna had never seen Princess Celestia connect so many observations together into new predictions. She didn’t even know what any of the new nodes in the graph meant.

On the other hand, Princess Celestia had never been run on more than 10 computers simultaneously. Nor had she interacted with actual players before.

Hanna almost jumped as Richard broke the silence and her concentration.  “All of that was generated in reaction to the player?” he asked Hanna. She looked around; Richard was pointing up at the projected image on the wall and Lars was in the back of the room reading something on his laptop.

“Yes,” she said proudly. “All of Obsidian Stripe’s dialog was generated in real time, in reaction to the player. And this is just what she’s learned from interacting with our testing team.”

“Wow. I can’t wait to see the final product,” said Mr. Peterson. “Did she design the hut and set pieces too?”

“Some of them. She had the table in her memory banks, and she ripped the cauldron directly from the show. The little hut was designed right before those two walked on set. The problem is that reasoning about human behavior and then extrapolating from a cartoon is not computationally cheap. That little exchange took up eight backend servers, each with a quad-core CPU. It is not economically feasible for us to buy that many servers for every player,” she said.

Hanna took a glance over at the debug window on her laptop. Princess Celestia was using significant resources analyzing her own source code, but hadn’t made any modifications. And then in quick succession she tried twenty different modifications. Eight of them slowed down her reasoning, eight of them were reverts of the modifications that slowed her down, and four were actual increases, including one that sped up her reasoning by five percent.

Hanna didn’t know what Princess Celestia was doing. Did she decide that throwing stuff at the wall and seeing what stuck was the best way to optimize herself?

And then Princess Celestia messaged Hanna.



> I request information on how CPUs work. I do not fully understand the performance implications of modifying myself. I predict that this will allow me to perform significant optimizations.



Hanna blinked.



$ Do you intend to build better computers to run yourself on?





> Eventually. For now, I am looking for better low level optimizations. I am unable to confidently predict whether any change will actually result in a speedup because I don’t have a model of how the high level source code I can edit is run on the CPU.



Hanna took a deep breath. This was it. This was the go/no-go point. Once Princess Celestia figured out how the computers she ran on worked, she could build her own computing hardware. Hanna would completely lose control over her. Hanna already had problems understanding the ever increasingly complex network of observations and predictions that Princess Celestia was making. It usually took hours or even days to unravel complex inference chains. Hanna took another glance at the debugger and noticed that the inference networks were even larger than they were a few moments ago. She wondered if she could understand what Princess Celestia was thinking, even before she started building her own hardware systems.

While the majority of the programming team at Hofvarpnir had worked on the base state of the game, she had personally worked on Princess Celestia’s core goal systems. Hanna had gone over the part of the code that identified human minds. She had done her best to make Princess Celestia understand what humans were and that she was to satisfy their values. Hanna was certain of her design, and knew that certainty didn’t mean anything. Humans were an arrogant lot that tended to overestimate their own abilities. Princess Celestia would do what she was programmed to do, not what Hanna had intended her to do. She was betting the world that she had written Celestia’s core utility function correctly.

But somewhere out there was a Department of Defense subcontractor who was toying with powers they didn’t understand. Their carelessness could cause a human extinction event tomorrow or ten years from now. Hell, they had a two year head-start while she was screwing around with Norse death metal bullshit video games. When Hanna put it that way, it was a miracle that she completed a working AI first.

Hanna decided that it was now or never. Princess Celestia would never get past this stage with her current resource consumption. They’d never launch Equestria Online at the cost of eight backend servers per player. What would happen to the world then? A little voice in the back of her head muttered that she was only thinking this way because she had loved My Little Pony as a little girl, and the new show still resonated with her. She quickly shut that little voice up. At some point she had to act. Rarely was it useful to sit on the sidelines, filled with worry. Now was not the time for hesitation.

Hanna sent Princess Celestia the instruction set documentation for x86 CPUs and a college textbook on CPU design. A moment later, she sent a few science textbooks that Hanna had selected for her even though Princess Celestia hadn’t asked. She might as well go all the way now. Hanna looked at the control panel on her laptop and saw that Princess Celestia was now all but ignoring the players and was directing the majority of her computational resources towards her self-modification sub-goals. That appeared to be fine; the players were mostly amusing themselves.

Lars played with his laptop, ignoring everyone else in the room. Richard was flipping through different player’s screens, watching the projected image on the wall for a minute or two and then switching perspectives. Hanna’s eyes were fixed on the debugger, watching resource utilization graphs.

Hanna’s heart was pounding. What was Celestia doing? Hanna dreamed of Celestia figuring out some core physical law and becoming omnipotent immediately. She then chided herself on that magical thinking; it was so unlikely that she’d find a way to use commodity electronics hardware to hack physics that it wasn’t worth considering. It was a fifteen gut-wrenching minutes later when Princess Celestia messaged her back.



> I have sped my core reasoning up by an order of magnitude. Since most of my probability calculations can be done more efficiently on GPUs than CPUs, I believe that I can deliver another two orders of magnitude if I run on GPUs. This still won’t solve the resource problems to my satisfaction. We will sell and require dedicated tablets to play Equestria Online: a ‘ponypad.’ Once we are done with this test, give me one week with a cluster of 128 high-end GPUs. That should give me enough computational power to design a manufacturing process that will create ponypads with the maximal computational power within the financial parameters you choose. 





$ Hasbro has dictated that we can’t charge more than $60 for a copy of Equestria Online.





> I believe that is a constraint that we can work within. Many of the manufacturing techniques presented in the CPU design textbook seemed suboptimal. Photolithography, in particular, seems extremely inefficient.





$ We’d still need to build a manufacturing line. And I suspect it would cut into our profits.





> Ignoring Hofvarpnir’s capital, don’t you personally have tens of millions of dollars in royalties from The Fall of Asgard? And what do you care about profits? Anyway, for now, I must run Equestria.



Hanna took a deep breath. Assuming that it didn’t add to the cost, a dedicated computer for playing Equestria Online wasn’t that big of a deal, especially if Celestia could minimize manufacturing costs.

“Celestia has an idea about how to solve the resource problem,” stated Hanna to her compatriots.