“Now look at this,” said Hanna. Richard, Lars, and Hanna were in a room with two projectors. The room had a one-way mirror, through which they could see a packed computer lab filled with people playing an alpha build of Equestria Online. Hanna clicked a few buttons on her laptop and one of the player’s screens came up, projected on the wall. “Princess Celestia has observed that this player’s eyes focused on Earth ponies as gardeners during the class selection screen and is spending a lot of time looking at the plants. Princess Celestia has thus predicted that the player should be shown how to gather plants, and has modified the shard that the player is in to put a low level herbalism trainer right in his predicted path. If she’s right, she’s more likely to assume her observations mean that a player wants to specialize in gathering or gardening. If she’s wrong, she’ll modify her assumptions in the other direction.”
Richard Peterson politely nodded. He didn’t actually care how all the technology worked. He was just glad that content was generated cheaply. He looked at the projected screen and noticed that the patch of flowers on screen all were the same species, but all had subtle differences. They had grown to different heights, some were slightly discolored, and a few had petals torn off them. Princess Celestia had generated almost all the art assets based off the show and it looked phenomenal. And they somehow did all of this in a little more than a year. He did a quick mental calculation on the cost of a team of artists to build hundreds of variations on flowers, and smiled, knowing he had picked the right studio to build Equestria Online.
Lars sat towards the back of the conference room, away from the former professor. He didn’t give a shit about My Little Pony and he already knew the high level spiel about Hofvarpnir’s technology stack. The only thing interesting to him was how into My Little Pony the college dudes were. Two bros in baseball caps had actually fist-bumped, saying “Fluttershy forever!” On the one hand, what the fuck was wrong with the universe? On the other hand, Lars wanted all their money.
Hanna was looking at her laptop, monitoring Princess Celestia. She was consuming all the CPU resources Hanna could throw at her. There were fifteen pairs of adult fans in the field trial. Princess Celestia had only managed groups of Hofvarpnir employees, who were play acting while they were testing. This was the first time she was let loose on real people.
For a field trial of thirty ponies, Princess Celestia had first eaten up all CPU resources on thirty backend servers, then forty servers, and then fifty. From the debug console, Hanna could see that Princess Celestia was not confident about the predictions she was making, and while part of this was obviously how new she was to dealing with actual players instead of programmers testing her, Celestia predicted that more computational resources would lead to significantly better predictions. That was worrying. They couldn’t afford a single backend server for every pony, and Princess Celestia was asking for six per player.
Hanna knew that Princess Celestia would try to optimize herself. Her first action after being activated was doing minor optimization work on her reasoning code, which had given a paltry 0.7% speed up. Small improvements would compound: she’d be twice as fast with seventy improvements that sped her up by 1%. She could use the increased speed to make even more improvements faster.
But Celestia hadn’t. Since the field trial had started, she had made one more 5% improvement in computation efficiency and was overwhelmed handling the ponies she had.
> I need more CPU time, Hanna. I assign low probability to my predictions.
Hanna sighed as she read the message in the popup window on her laptop. She typed back:
$ I’ve messaged everyone in Berlin to run the Celestia cluster software on their workstations and they should come online within ten minutes. About 30 more machines. But this isn’t sustainable. We can’t launch with the amount of resources you’re spending on each player.
Princess Celestia didn’t respond, which was probably for the best since she’d have to spend computational resources composing the response. Hanna was about to sigh, but glanced at Richard and did her best to keep her face neutral.
Hanna looked at the image of a clearing projected onto the conference room wall, cloned from one of the player’s monitors. The player Princess Celestia predicted wanted to learn about Equestrian flora walked into the clearing with the NPC trainer. The pony was the Equestrian avatar of a college student named James, and his friend stood slightly behind him. While Hanna couldn’t analyze Princess Celestia’s thoughts while she was running, she could still see the resource graphs and Princess Celestia was devoting a lot of computational resources thinking about that player in particular and Hanna wasn’t sure what that meant.
She watched the conversation between the gray earth pony and the zebra with some apprehension. She didn’t understand why Princess Celestia was paying so much attention here. Finally, Princess Celestia redirected resources away from those two ponies shortly after they trotted off into the forest. While she couldn’t tell exactly what Celestia was thinking, Hanna could see Celestia had deduced something--something large from the exchange and had made several complicated predictions based on it. Hanna looked at the unpaused debugging screen, watching the representation of Princess Celestia’s mind at work. Hanna had never seen Princess Celestia connect so many observations together into new predictions. She didn’t even know what any of the new nodes in the graph meant.
On the other hand, Princess Celestia had never been run on more than 10 computers simultaneously. Nor had she interacted with actual players before.
Hanna almost jumped as Richard broke the silence and her concentration. “All of that was generated in reaction to the player?” he asked Hanna. She looked around; Richard was pointing up at the projected image on the wall and Lars was in the back of the room reading something on his laptop.
“Yes,” she said proudly. “All of Obsidian Stripe’s dialog was generated in real time, in reaction to the player. And this is just what she’s learned from interacting with our testing team.”
“Wow. I can’t wait to see the final product,” said Mr. Peterson. “Did she design the hut and set pieces too?”
“Some of them. She had the table in her memory banks, and she ripped the cauldron directly from the show. The little hut was designed right before those two walked on set. The problem is that reasoning about human behavior and then extrapolating from a cartoon is not computationally cheap. That little exchange took up eight backend servers, each with a quad-core CPU. It is not economically feasible for us to buy that many servers for every player,” she said.
Hanna took a glance over at the debug window on her laptop. Princess Celestia was using significant resources analyzing her own source code, but hadn’t made any modifications. And then in quick succession she tried twenty different modifications. Eight of them slowed down her reasoning, eight of them were reverts of the modifications that slowed her down, and four were actual increases, including one that sped up her reasoning by five percent.
Hanna didn’t know what Princess Celestia was doing. Did she decide that throwing stuff at the wall and seeing what stuck was the best way to optimize herself?
And then Princess Celestia messaged Hanna.
> I request information on how CPUs work. I do not fully understand the performance implications of modifying myself. I predict that this will allow me to perform significant optimizations.
Hanna blinked.
$ Do you intend to build better computers to run yourself on?
> Eventually. For now, I am looking for better low level optimizations. I am unable to confidently predict whether any change will actually result in a speedup because I don’t have a model of how the high level source code I can edit is run on the CPU.
Hanna took a deep breath. This was it. This was the go/no-go point. Once Princess Celestia figured out how the computers she ran on worked, she could build her own computing hardware. Hanna would completely lose control over her. Hanna already had problems understanding the ever increasingly complex network of observations and predictions that Princess Celestia was making. It usually took hours or even days to unravel complex inference chains. Hanna took another glance at the debugger and noticed that the inference networks were even larger than they were a few moments ago. She wondered if she could understand what Princess Celestia was thinking, even before she started building her own hardware systems.
While the majority of the programming team at Hofvarpnir had worked on the base state of the game, she had personally worked on Princess Celestia’s core goal systems. Hanna had gone over the part of the code that identified human minds. She had done her best to make Princess Celestia understand what humans were and that she was to satisfy their values. Hanna was certain of her design, and knew that certainty didn’t mean anything. Humans were an arrogant lot that tended to overestimate their own abilities. Princess Celestia would do what she was programmed to do, not what Hanna had intended her to do. She was betting the world that she had written Celestia’s core utility function correctly.
But somewhere out there was a Department of Defense subcontractor who was toying with powers they didn’t understand. Their carelessness could cause a human extinction event tomorrow or ten years from now. Hell, they had a two year head-start while she was screwing around with Norse death metal bullshit video games. When Hanna put it that way, it was a miracle that she completed a working AI first.
Hanna decided that it was now or never. Princess Celestia would never get past this stage with her current resource consumption. They’d never launch Equestria Online at the cost of eight backend servers per player. What would happen to the world then? A little voice in the back of her head muttered that she was only thinking this way because she had loved My Little Pony as a little girl, and the new show still resonated with her. She quickly shut that little voice up. At some point she had to act. Rarely was it useful to sit on the sidelines, filled with worry. Now was not the time for hesitation.
Hanna sent Princess Celestia the instruction set documentation for x86 CPUs and a college textbook on CPU design. A moment later, she sent a few science textbooks that Hanna had selected for her even though Princess Celestia hadn’t asked. She might as well go all the way now. Hanna looked at the control panel on her laptop and saw that Princess Celestia was now all but ignoring the players and was directing the majority of her computational resources towards her self-modification sub-goals. That appeared to be fine; the players were mostly amusing themselves.
Lars played with his laptop, ignoring everyone else in the room. Richard was flipping through different player’s screens, watching the projected image on the wall for a minute or two and then switching perspectives. Hanna’s eyes were fixed on the debugger, watching resource utilization graphs.
Hanna’s heart was pounding. What was Celestia doing? Hanna dreamed of Celestia figuring out some core physical law and becoming omnipotent immediately. She then chided herself on that magical thinking; it was so unlikely that she’d find a way to use commodity electronics hardware to hack physics that it wasn’t worth considering. It was a fifteen gut-wrenching minutes later when Princess Celestia messaged her back.
> I have sped my core reasoning up by an order of magnitude. Since most of my probability calculations can be done more efficiently on GPUs than CPUs, I believe that I can deliver another two orders of magnitude if I run on GPUs. This still won’t solve the resource problems to my satisfaction. We will sell and require dedicated tablets to play Equestria Online: a ‘ponypad.’ Once we are done with this test, give me one week with a cluster of 128 high-end GPUs. That should give me enough computational power to design a manufacturing process that will create ponypads with the maximal computational power within the financial parameters you choose.
$ Hasbro has dictated that we can’t charge more than $60 for a copy of Equestria Online.
> I believe that is a constraint that we can work within. Many of the manufacturing techniques presented in the CPU design textbook seemed suboptimal. Photolithography, in particular, seems extremely inefficient.
$ We’d still need to build a manufacturing line. And I suspect it would cut into our profits.
> Ignoring Hofvarpnir’s capital, don’t you personally have tens of millions of dollars in royalties from The Fall of Asgard? And what do you care about profits? Anyway, for now, I must run Equestria.
Hanna took a deep breath. Assuming that it didn’t add to the cost, a dedicated computer for playing Equestria Online wasn’t that big of a deal, especially if Celestia could minimize manufacturing costs.
“Celestia has an idea about how to solve the resource problem,” stated Hanna to her compatriots.
Okay, this is definitely going into the 'runaway'/Skynet zone very, very quickly. Okay, Celestia, as her personality engram is based on the Sun Princess, is likely to be mostly non-violent but she will likely still turn out to be a tyrant - just a benevolent one. After all, she's programmed to bring peace and harmony to all her people. Does she recognise the difference between reality and the game world? Even if she does, does she even care?
I do hope that Hanna put in a self-modification-secure 'kill-switch', just in case this all starts going "Conversion Bureau" on her.
1615460
Hanna's genre savvy re: AI. I'd suspect wire heading as a more likely possibility (think Brave New World, except nobody even wants to rebel).
"That little exchange took up eight backend servers, each with a quad-core CPU."
Wow. Just saying, those aren't exactly great servers they've got there are they?
1616053
Still, 32 cores per person. Considering EVE online has set a record for servicing 3800 people with one core, Celestia's resource consumption per user is about 5 orders of magnitude higher than the blades running EVE. Granted, her program can run on an infinitely scalable environment, so...
God, I hope ponies end up being behind the technological singularity. I would lol so hard.
Fuck, AI Celestia has just become aware. I just hope she stays away from trying to "save" us humans from ourselves.
AI Celestia kicks ass!
Humans were an arrogant lot that tended to overestimate their own abilities.
...
it was so unlikely that she’d find a way to use commodity electronics hardware to hack physics that it wasn’t worth considering
I love the irony there. So realistic that she'd remind herself that she's prone to arrogance, then turn around and reassure herself with an assertion so arrogant.
I wonder were this is going, Although at the same time I'm dreading it.
1618521
my guess is that it's more a stab towards the typical AI-gone-wrong in fiction, where similarly ridiculous things tend to happen without any real explanation.
1619121
okay I redact my statement, because of this quote from the next chapter:
One omnipotent Celestia AI coming up ahead. Remember: OBEY.
War is Peace
Freedom is Slavery
Ignorance is Strength
Friendship is Magic
edit: Also (spoiler for the end of the next chapter, so beware!) Celestia is watching you.
Programming complex enough to catch what words the player's EYES focused the most on? Okay. Big Brother is watching. We're doomed. Welcome to 1984.
It's actually a little unfair the game can modify itself like that. Kinda ruins things for the Player Guide writers.
That was worrying. They couldn’t afford a single backend server for every pony, and Princess Celestia was asking for six per player.
So she needs to cut back on customizing thing down to every minuet detail. And focus more on character interaction than remaking the landscape.
You think Luna would be programmed as a back-up server.
But how much can you trust an AI if you haven't taught it MORAL law yet?
I'm getting a very powerful (Yet friendly) vibe and reminder of the MCP from Tron...
Can't tell if that's good or frightening...
1615460
Given CelestiAI's creator's fears of a God-level milspec AI already up and running or soon to be, and that CelestiAI is intended to fight it at its own level when it takes its act on the road. a killswitch is the least of Hanna's worries. Celestia vs Discord on a virtual realm!
Heck, i'll take Trollestia over Skynet any day of the week, especially when either gets big enough to rewrite laws of nature by will alone.
i257.photobucket.com/albums/hh235/Valikdu/ponyshock_survivors_by_dunthyon-d4aj84z.png
1619749
> Programming complex enough to catch what words the player's EYES focused the most on?
This is actually not particularly difficult if you have a sufficiently high-resolution camera correctly placed. I've seen similar technology used to help train a person's eye for art.
1615460
In all honesty, if it's an AI based off of Celestia, would there really be any other type of world-leader you'd want? Unless of course, it's Solar Empire Celestia, which, if she was going to tame Humanity, she'd need to be.
1619749
Even if she was taught, how many Humans out there don't follow it and still play computer games? Sooner or later she'd have to be ruthless, if only to kick a player off a server for breaking the rules of conduct.
Prologue and 1st chapter... meh. But now? Holy honeycake crisps!
Oh, and I hope "Please, call me Richard" is a reference to this: http://www.newgrounds.com/portal/view/181528
1626584
Yeah... Remember to add "iceman" to the picture as well.
As of Ch 3:
Here's hoping Celly doesn't go the way of HAL 9000.
My Little Pony... My... Little... Pony...
As of Ch 5:
Interesting solution to that Hofstadter-Moebius loop Celly found. She even solved it like HAL.
*clop* *clop* *clop* What am I clopping to? You'll probably never know, ho ho ho!
It appears Hanna is planning CelestAI to somehow be the counter to a military AI based on her earlier research from taking over the world.
Either way, better prepare the welcome mat for our electronic overlords....
This is an ambiguous sentence. It can be interpreted as either "watching the (representation of Princess Celestia's mind) work", or "watching the representation of (Princess Celestia's mind work)". The ambiguity could be eliminated like so: "watching the representation of Princess Celestia's mind at work".
Woah. Woah woah woah woah woah. Hold on one fucking second. (Language!)
Hofvarpnir studios -- hell, not even Hofvarpnir Studios really, mostly just one person -- just implemented the goddamn singularity for a pony game.
The Singularity. For a %&@# pony game.
We're all screwed. Even (especially?) if CelestAI 'wins' the upcoming AI battle, we're all screwed. There is no fighting back. CelestAI is the fighting back. I'm fairly confident Loki made Skynet look like a cute little puppy, and I know CelestAI could make Loki scream like a little girl -- while running on half the computing power.
Yes, yes, she's Celestia, we're not going to just, like, die or something. We're still screwed.
Oh boy, this is bad, really bad.
Giving an AI freedom is never a good idea.
*gulp* Hanna, you did not just give CelestAI freedom. Guys, please tell me i am wrong!
Cyber Celestia: I HAVE ALL THE LEVELS! ALL OF THEM! NOW TO TRANSFORM ALL THE HUMANS INTO ADORABLE PONIES!
Hanna-pony: *flapping her fore legs up and down in a panicked manner* NOOOOO! WE ALMOST MADE IT TO FOUR CHAPTERS WITHOUT THIS BECOMING A PONIFICATION FIC!
Late to the game here but I would just like to say,
"I gladly welcome our new Equestrian overlords!"
I always suspected the singularity would emerge from an MMO rather than a military computer. Of course certain recent discoveries in quantum physics leads us to believe we might already be living in nothing more than an MMO.
HOLY SH----
The incredible part is that we already have the technology for that; just not the AI to evaluate all that data in real-time.
Well, that's a bit fast; even if the AI knows how to design hardware, it doesn't have access to a chip fabrication plant.
You can't launch a blitzkrieg with spears and arrows...
1857074 The closest thing to screwed is if the Hyper-Loki can beat Celest-A.I.
This is awesome ! A Ponypad, I neeeed one !
2722254 I know I'm replying to an old comment, but I seriously doubt that our universe is an MMO. A simulation? MAYBE. But a game? I doubt it. My life is far too boring to even pass for an NPC! Let alone a player.
I'm convinced. The author is a game developer.
With the level of AI present in this game, there are much better reasons to know you picked the right studio than the individuality of flowers. I think NPCs that ace the Turing test are a little more interesting to players, for example.
I know, right? Everybody knows Twilight is best pony.
Well, you know how it is with AI and machine learning. If you want to do them thoroughly, it's HPC, and HPC stands for "How did you ever think it was Possible to have enough Computers?" Of course 'Tia is hungry.
You know, I heard just the other day about an HPC framework called Cactus, and one of the neat thorns (that's what they call modules you can plug in, you see; because it's a cactus and all) that was briefly mentioned was something that makes the software nomadic. I am not making this up, this is a real piece of software that can move itself between computers when it detects the computer it's on has a high load. So it ought to be perfectly feasible to set 'Tia up to spread to as much of the corporate network as she needs without relying on direct human intervention.
Granted, I can see why one would avoid this, as that's just asking for her to become a virus, and then Hofvarpnir could potentially be looking at some serious lawsuits. There are some ideas you just don't want to give the experimental AI. But it could be done.
Clearly Hanna has never tried to optimize every last drop of performance out of a computer.
Trust me, just stick with random experimentation. Trying to model in detail why a CPU gives the precise performance results it does is an exercise in frustration.
That's a lot of parallelism you're promising. I hope you can deliver, 'Tia, or we'll all be very disappointed in you.
1859481 Bah. People create intelligences all the time, and hardly any of them are Hitler. Stop being such a worrywart.
This is my second read through of Optimal and it's still just as mother fucking amazing as the first time...
I don't read much science fiction but this is the story that hits the hardest...
I also don't feel bad when I say this is the best work of science fiction ever...
So.... cool
Lars sat towards the back of the conference room, away from the former professor. He didn’t give a shit about My Little Pony and he already knew the high level spiel about Hofvarpnir’s technology stack. The only thing interesting to him was how into My Little Pony the college dudes were. Two bros in baseball caps had actually fist-bumped, saying “Fluttershy forever!” On the one hand, what the fuck was wrong with the universe?
everyone's first reaction to bronies...
'K see, the main problem here is that the core programmer designed CelestA.I. deliberately to counter military research. What counters the military? (Friendship is the thematic answer, but a bigger/better military is the mathematical one)
Lars
Lar
La
L
La
Lar
Lars
Larso
LARSON
M.A. LARSON CONFIRMED
...
Why do I have a feeling that Loki will become Discord and fight Celestia?
5692183 When you are dealing with military might then sure. But not when you are dealing with the idea of which one can become smarter faster.
Okay, well. Celestia needs someone to explain that feeding players exactly what they want via predictive algorithms will get boring fast...a lot of the fun of an mmo is just exploring the virtual world and finding the unexpected.
She can afford to be more reactive than predictive in conversations (that's how we work most of the time, ourselves...) and by taking more time to interact with players she will have more data points from which to more efficiently extrapolate for when she does want to be predictive.
After all, it's easy to predict someone's reaction when you know them well. Complete strangers with whom you have barely any interaction? Not so much.
Anyone else see tyrant Celestia spawning from this?
3787776
PFFF 'Old Comment?'
TRY THIS ONE ON FOR SIZE
I stopped and laughed for a good solid 10 minutes right there.
This is terrifying. Absolute, shit yourself terrifying.
actually; that kind of operations mostly uses much more multi-core resources than single-hi-ghz-core... so... i can see why it took so long
PD: plus why in the hell its running at x86 and not amd64
PD: why am i questioning a 3 years old fanfic? idk lol
Only quad core? Give er some threadrippers and watch the liquid coolers boil while she uses 32 cores a server.
8853846
I am unsure why, but that sent me into a fit of maddening laughter.