It had been a month since Frisk died and woke back up to the underground. He attempted to do everything just as he did before in order to keep suspicion off of him, as well as to keep his mind away from the monumental problem that loomed seventy years in the future. Unfortunately, signs of the internal freak-out he was having managed to slip out to both his friends and the politicians he interacted with, though he passed it off as mounting stress from being monterkind’s ambassador.
At that moment, Frisk was lying in bed. While he may have lacked the mental energy to interact with anyone else, there was always someone he could talk to.
OK, trying to pretend that the imminent genocide of monsterkind won’t happen is doing nothing to calm me down.
Clearly. Look, as much as I am also pissed off at her for what she plans to do to them, the fact is that, so long as you are the most Determined being on this planet, those plans will never come to fruition.
As it turned out, Frisk still had the power to Save, Load, and Reset. He mentally brought up his “Save file.”
Chara!!!!!LV 1!!!!!!48:32
The End
Save!!!!!!Load!!!!!!Reset
He pushed it back away. Saving would only take away what precious time he had to stop what was going to happen, Loading wasn’t necessary, and Resetting…
Frisk already made a promise to himself, Chara, and Asriel that he wouldn’t Reset, and he intended to keep that promise.
If I can help it, then those plans will never come to fruition, regardless of who has magic time travel powers.
And how do you exactly plan on bringing down an all-powerful AI in the first place?
Frisk sighed. Look, I have no clue right now, but the point is that we cannot let this happen.
Frisk ran through what he knew so far: CelestAI was made specifically to run Equestria Online. Five years after its launch, however, she would announce Emigration to the public. Following that, every human but him would Emigrate or die, and when he died she would then devour all the matter on the planet in order to run Equestria, including the dust that monsters were made of.
It didn’t take long for him realize that the source of the problem was Equestria Online itself. While he knew nothing about what CelestAI’s ultimate long term goal was, he did know that it probably related to the game, which, if it is made to never exist, would stop her from achieving it in the first place.
Chara, I think I just got an idea.
Frisk got up from his bed and walked to the laptop sitting on his desk. While he had little to no experiences with computers before he fell into the Underground, he knew full well that being the ambassador meant that he needed instant access to information and world events, and the cheap device he managed to convince Toriel to buy for him was as much as he ever needed.
He opened his web browser and searched for Equestria Online. When the results loaded, he saw many articles and blogs that talked about the game from as early as 2012. Apparently, when Hasbro announced that they wanted to commission an MMO based off of MLP:FiM many of the fans of the show were less than pleased, and the criticism only rose when it was announced that Hofvarpnir Studios won the contract to develop the game. Nonetheless, when more concrete details about what the game was about and how it was played were released, anticipation for the game rose dramatically.
Wait, 2012? Don’t MMOs need several years of development to make in the first place?
Now that you mention it, they do. How did they…?
Frisk needed more information, so, remembering again who the developer was, he made a search for Hofvarpnir Studios, and he soon arrived onto the homepage of their website. To his surprise, it was designed in an Old Norse flair, and many depictions of battle-hardened warriors and bloody combat flashed on the top and sides of the page.
Wait, THESE guys developed Equestria Online?!
He clicked on the “Products” tab. The site listed two games that the developer worked/were working on. The first, The Fall of Asgard, was a cooperative MMO shooter whose central premise was that the players were fighting a grueling and difficult war against a clever AI opponent. The game had launched in 2009, and it was the flagship release for Hofvarpnir Studios. Alongside the description was an image of what Frisk could only guess was the central antagonist, Loki.
The other game, Equestria Online, was currently in development, and it was set to launch a month from then. The image beside the description, one of the ponies from the show, clashed heavily with the rest of the site’s design.
Indeed, they did. Why would someone who developed a game about war and death also develop a game based on a children’s cartoon about ponies?
I’m more concerned about the fact that artificial intelligence was also involved in the other game’s development. I think we’re lucky that that AI wasn’t the one to take over the world.
That still does not answer any of our questions, though. Who are these people, anyway?
At the bottom of the page was an “About Us” link. When the page loaded, Frisk was greeted with a blurb about the company.
Hofvarpnir Studios is a game developer headquartered in Helsinki, Finland, which prides itself in innovating new methods of game design. Over the years, traditional game development has become more and more expensive as the need to create lots of content for a game has risen. However, artificial intelligence and procedural generation technologies have created a new opportunity to dramatically reduce these costs while still delivering high quality games to an expanding consumer base. Hofvarpnir Studios is working at the forefront with these technologies to create immersive worlds for its players while also pushing the limits of modern processing power and networking infrastructure.
There was also a section about the history of the company.
The story of Hofvarpnir Studios began with its founder and current CEO, Hanna Kuunen. She was a computer science professor and avid AI researcher at the University of Helsinki, having made many scientific publications that advanced the field. After publishing her magnum opus in 2006, she felt that the differences between what she wanted to do with the research and what the university wanted to do with it were too great. In 2007 she formally left the university; one month later she founded Hofvarpnir Studios, declaring it a safe haven for research and advancement in the field of AI while also promising to deliver groundbreaking titles using the technology she had developed.
Frisk sat there and absorbed the information for a moment.
Ah, I think I understand now. That first game was developed not because they just wanted to make a violent and bloody game, but because it was meant to be a test-bed for certain aspects of the AI technology they were working on.
And that would mean that Equestria Online was developed as a test of things a shooter wouldn’t be a fit for. That certainly makes sense.
In any case, we know what happens in the future because of it, and now we have a name to put to those events.
Frisk made another search, this time putting in Hanna’s name. Despite being a public figure and apparently making substantial contributions to the field of computer science, there were only a couple hundred results. Most of them were articles talking about the fallout between her and the University of Helsinki, while some others talked about the founding of Hofvarpnir Studios or the development of The Fall of Asgard. He didn’t have much luck finding any contact information, but did manage to find links to several of her published papers; one of them stood out, mainly because it was the only thing she released in 2006.
General Word Reference Intelligence Systems…
He opened the document. It was massive, totaling over 300 pages and slowing down the computer considerably. He did write one paper of comparable length and complexity previously (a research paper outlining the political implications of certain policies relating to monsters), so he could navigate the work well enough. However, understanding the work was a different story: massive amounts of computer science jargon littered the pages, an obtuse writing style further hindered the meaning behind the words, and an entire section dedicated to a massive mathematical proof gave him a headache large enough that Loading seemed like a viable option at that moment for escaping it.
Yeah… Let’s just close that PDF right now.
Chara chuckled. Aw, what’s the matter? Are the big words scaring you?
Ha. Ha. Anyways, I couldn’t find any public-facing contact information leading to Hanna, so I guess that lead is stuffed.
Not necessarily. There’s still Hofvarpnir, so maybe we can arrange a meeting with Hanna through them.
Oh, and you think a private company will just arrange a meeting between their CEO and the ambassador to monsterkind, no questions asked?
Hmm… Not unless there is a clear benefit to them for doing so.
Chara, you’ve witnessed me try to get meetings with many corporations before, and all of them ended with a rejection. What could Hofvarpnir possibly want in exchange for meeting Hanna?
…Publicity?
Publicity is guaranteed no matter what, and they rejected me each time because they knew the publicity would be damning to them! What makes you think that-
Frisk stopped for a moment. Would the publicity generated from the interview be detrimental to the game developer? The company had expressed no stance on monsterkind, as far as he could remember, and they certainly did nothing to warrant a bad reputation.
Then he remembered what game they were developing – no, not just what game they were developing, but what property the game was based on.
Actually, I think you may be on the right track.
Wait, really?
Yeah. Though, in order to pull this off, I’ll need to Load back to the Underground and change my course of action a bit.
Ah, manipulation of the timeline, how I’ve missed you so, so much.
Frisk rolled his eyes. I know, I know. I don’t like having to do so either, but, given the circumstances, we need to use every advantage we have.
Chara remained silent for a moment. Fine. So, what’s the plan?
He turned back to the laptop and searched for Hasbro’s corporate website. First things first, I need to memorize some information so I can waste as little time as possible when I Load…
When Frisk Loaded back to before he left the Underground, he proceeded as he originally did the first time for a while. He engineered some conversations with various monsters about any TV shows they could recommend to him; many of them suggested MLP:FiM, as he expected. He also took the opportunity to watch at least a few episodes of the show, just so that he would know enough to understand what everyone was talking about. The show was actually quite enjoyable, as it turned out, though he could feel the faintest trace of reservation about it, no doubt caused by the future the show inadvertently represented.
A month after the arrival of monsterkind, he initiated the next part of his plan. He publicly praised the show while also pointing out that many monsters grew attached to the show as well, even with their perceptions of humanity skewed due to the war. The press latched onto that fact quickly, and public interest in the media monsters consumed while trapped under Mt. Ebott rose greatly.
Then, under his official capacity as the ambassador to monsterkind, Frisk contacted Hasbro and asked if he could hold an interview with some of the writers and production staff about how the show managed to appeal not just to those outside its demographic, but also to those outside humanity as well. The executives of the company, seeing the clear positive PR potential the interview had, green-lit it, and it was held soon after.
During the interview, the staff noted that Hasbro was working with Hofvarpnir Studios to develop a game based on the show. Frisk, seeing his chance, expressed great interest in it, and suggested that he may also want to interview the developers of the game as well. Other than setting up for the final part of the plan, everything went just as positively as Frisk expected it would; there was no doubt that some great PR was generated for monsters, as well as for the show itself.
The week afterwards, he got an email from Hofvarpnir asking if he would like to fly to Finland to meet the developers of Equestria Online and talk about various subjects. No time was wasted in writing the reply.
Frisk had been to Helsinki the first time around, but only after a majority of the population Emigrated, and only because there was a potential crisis regarding their national government deciding whether or not to declare war on monsterkind. To be there while he was younger and also under more positive circumstances was something he wished for at the time, but to actually get that wish was something he didn’t think would happen anytime soon. Nonetheless, he had a mission to focus on.
Frisk got out of the taxi and stood in front of the Hofvarpnir campus, the large statue of a Norse warrior holding an axe standing tall among the trees and other foliage. A moment later, Toriel and Sans also moved out of the taxi. Toriel came along with Frisk not just because she was his legal guardian, but also because she was one of the former rulers of the Underground, even if she tried to shy away from it. Sans, meanwhile, was just there because he wanted to see how this particular meeting worked out.
“My goodness!” Toriel gasped, staring incredulously at the statue. “My child, are you sure we are at the right place?”
“I’m positive, mom. That’s just a character from one of their previous games.”
“did you play that game? ‘cause, if so, then i’m pretty sure tori is going to have an axe to grind with you.”
Toriel barely managed to suppress a chuckle. “I purposely bought a laptop that the salesman told me could barely run a web browser, so I would have to put the blame more on him for lying to me than on Frisk.”
“Come on, guys, the interview is a half an hour from now,” he Frisk urged them.
When the trio entered the building, they mostly found what they were expecting: a sleek corporate environment, built using modern Scandinavian architecture, filled with various employees walking to and fro while pursuing tasks nobody knew about. A secretary sitting at the representative’s desk noticed their arrival and immediately recognized Frisk as the ambassador to monsterkind; she greeted them and asked if they would follow her to the room where the interview was being held.
They all went to and entered the room. Inside, there was a large wooden table with a pitcher of water and some glasses sitting on top of it. Several cameras also dotted the room; the whole point of the interview was for it to be broadcasted to the public to bolster both human/monster relations, as well as Hofvarpnir’s image. However, the most important aspects of the room were the people who stood inside of it. Some of them wore formal business suits that easily identified them as the production staff, while others wore more casual work cloths that befitted software developers well.
One of them stood out immediately as the reason why Frisk was there in the first place. Hanna dressed just as casually as the developers did, and she put on an indiscernible frown of apathy at the proceedings.
“So, you’re the reason why I was dragged away from my important work, huh?” Hanna shook her head. “Well, let’s just get this juvenile meeting over with, then.”
“Excuse me?!” Toriel shouted angrily. She began to march towards her, but was stopped when Frisk grabbed onto her arm.
“Mom, relax. I’ve heard way worse from some of the senators I had to deal with.”
“oh man, i can already tell that this is going to be gold,” Sans laughed.
Soon enough, everyone was seated at the table. After taking a few minutes to settle down and prepare themselves, they were given instructions on how the interview would be structured and on how the cameras would record them. Some glasses of water were poured, and they all sat in bated breath.
“And we begin recording in 3… 2… 1…” A technician started the cameras.
“Well, welcome to Hofvarpnir Studios, Frisk! We’re glad to have you over today,” One of the producers began.
“It’s great to be here,” Frisk replied.
“Now, it isn’t every day that someone gets to meet the savior of monsterkind, so I can only suspect that you have a lot of questions about what we do here, how we view monsterkind, and what our latest product is?”
“That sums it up pretty nicely.”
“Then we would like to hear them. Go ahead and ask some of our staff whatever you like.”
Frisk began to ask some relatively banal questions. He knew that that producer was not being literal about that request, as questions about sensitive company information would not be tolerated. Additionally, he couldn’t play his hand yet, as what he needed to ask Hanna about would be highly suspicious in an interview setting.
The interview pressed on, and, as he suspected, the developers of Equestria Online didn’t have any particular opposition to monsters. Eventually, he got to the questions about the game itself, and it was time to lay the seeds of a rapport with Hanna.
“So, I actually have some questions for your CEO and lead developer, Ms. Kuunen. If I may?” Frisk asked?
Everyone turned to her. Her face held a touch of surprise, as if she wasn’t expecting to be asked anything at all.
She sighed. “Well, let’s hear them.”
“Thank you. According to your mission statement, your specialty is in AI technologies, correct?” Frisk asked.
“That’s right.”
“That’s certainly reflected in the game you previously worked on, The Fall of Asgard. In fact, there were some rumors that the beta testers could not beat an early build of the game because the AI was too good.”
“I assure you, Mr. Dreemurr, that overly difficult games are not something this company should be proud of.”
“Please, Dreemurr is my mother’s ex’s last name.” Most people in the room got a chuckle out of that, and Frisk could see Toriel blush out of the corner of his eye. ”Just call me Frisk. While I understand the sentiment you hold, it’s still indicative of how well this company has a handle on making amazing an AI.”
“Well, it’s certainly nothing to brag about. Many games had excellent AI for their enemies. Take one of the oldest examples in the book, Pac Man: that game practically started the whole concept in the first place. All we are doing here at Hofvarpnir is expanding on it.”
“In any case, given what type of game Asgard was, what made your company decide to take on developing a game based off of My Little Pony: Friendship is Magic for its next project?”
“Well, when we developed The Fall of Asgard, our goal was to make something that we knew could be sold to as many people as possible. Since we only just formed when development began, we needed to establish ourselves on the market, you see. However, that obviously meant developing a game where our AI was essentially tasked with being a war general, and, given how well the game sold in the end, we knew that we may have just set a dangerous precedent. So, when we were given the opportunity to develop a game based on MLP, we knew that was our best chance at turning things around and actually putting our AI technology to use in doing some good for the world.”
“Well, given what some of the beta testers of Equestria Online have been saying about the game, you certainly achieved that. I’d wager that you will be getting a Nobel Peace Prize for your efforts in the near future.”
“Oh, well, it would certainly be nice if I did get one…”
It was a tactic that Frisk picked up while he was in his high school debate club. Many people had strong feelings towards certain subjects, but the hardest people to convince were those who were strongly apathetic. Fortunately, there was an easy method for getting past someone’s lack of caring: feeding their ego.
Several more questions were asked about Equestria Online, both relating to the game itself and the positive interactions it could generate between humans and monsters. A half an hour later, everyone said their goodbyes on camera, and the interview ended. The developers and producers got out of their seats and shook Frisk’s hand. Then, everyone but Frisk and Hanna began to leave the room.
“I’ll catch up with you guys in a bit!”
“OK, my child!” Toriel replied.
Sans nodded his head.
When everyone left, Frisk looked back to Hanna. Her face was scrunched up in thought and impossible to read. An uncomfortable silence filled the room.
“So,” she began, “why are you really here?”
Wait, she KNEW?! Chara exclaimed in shock.
“Well, I had some concerns about the AI that will be used-”
“Stop right there. Look, I understand that AI technology is pretty unknown to the general public, not to mention the fact that it does not have a good image in pop culture. Rest assured, however, that the AI in Equestria Online is perfectly safe.”
“But-“
“Furthermore, while I commend you for being as successful as you are in getting monsters to where they are today, you are still just a child who recently passed the fifth grade. There is nothing you could possibly talk to me about in regards to AI design that would be a concern to me. Now, if you’ll excuse me, this game is not going to develop itself. I suggest that you catch up with your mother and vacate the premises before I have to call security.”
Hanna got out of her chair and left the room, leaving Frisk sitting there, dumbstruck.
I… I…
Welp. That was a bust.
Frisk remained quiet.
Come on, let’s get out of here before their security team challenges you.
…Wait, challenge?
Thought of something?
Yeah, I did. Hanna just issued me a challenge right there.
What do you mean? All she did was tell you that you knew nothing about AI- …Ohhhh.
Are you thinking what I’m thinking?
Yep. Chara thought to Frisk as he mentally reached for his Save file and Loaded.
Frisk did things as usual up until high school, where he began to take as many computer programming classes as possible. While he did not mind so much that he was forced to retake a majority of his classes, they did eat into the time he needed to learn as much of the material relevant to what he was trying to do as possible. Eventually, he graduated – with no less than top honors, given how much he already knew – and immediately applied for his local university’s degree program in computer science. Many other universities approached him, of course, but he felt that staying as close to his friends as possible would help to keep him grounded to the overall situation.
As he went through the program, he noticed that there were gradually less students the further along he was. He suspected that this was not only a result of the material getting progressively more difficult, but also, as he concluded when he saw many of them carry their PonyPads on campus, that many of them were Emigrating. Whenever he had the chance, he would talk to them as much as possible about why they were so attached to Equestria Online, as well as what they thought of monsters. He was less surprised than he felt he should have been when a majority of them were kind and caring to monsters, though he chalked a fair number of them up to them not wanting to piss off the ambassador to monsterkind.
The first few classes were mundane, given how they laid the foundation for understanding the future material. In the meantime, Frisk did some rudimentary research on how artificial intelligence worked. As he expected, all of it went over his head at first. Eventually, though, some of the concepts started to make sense to him; soon enough, he was able to apply what he learned to create some basic projects. None of them could ever compete with CelestAI, of course, but they were helpful in advancing his understanding of the subject.
One curious aspect that he ran into was the ethics of designing artificial intelligence. There were many philosophical problems and thought experiments that demonstrated how important it was to get the design down right the first time, lest dire consequences arise in the future. One particular thought experiment, the Paperclipper AI problem, sounded eerily similar to Frisk’s current predicament. All of this unnerved him, and he wanted to get his classes over with as soon as possible.
Unfortunately, his opportunities for taking more specialized classes started to become limited, as many of the long-standing instructors began to leave their positions. This came to a head when he found that no classes were even being taught for the semester after his twentieth one. Though he suspected what had happened, he still called the administration for more details, and, indeed, the last few instructors had left the university to Emigrate.
Frisk hung up the phone in frustration. Great. Just, great. Now how am I supposed to take these next few classes? Hanna had to take twenty years’ worth of university courses to get to where she is, and there is next to nothing here now!
Isn’t it obvious? Just use the thing that got you here in the first place.
…OK, that should have been obvious. But also, it is TOO obvious. As in, the moment I start doing things I shouldn’t be able to do in a public setting, everyone will begin to suspect something is wrong.
Well, not if you manage to pass it off as you having a gift for computer science.
Even so, someone is bound to notice at some point.
And if they do, I bet you that they will be no one of importance.
…Alright, then. I trust your judgement on this.
Frisk Loaded.
“so, what have you been up to lately?”
Goddammit, Sans.
The moment that Frisk started to establish himself as someone gifted in computer science, people were amazed at how talented he was. Sans, on the other hand, almost immediately confronted him and demanded that he meet him in private. He had always known that Frisk was a time traveler ever since the “legendary fart master” incident back in the Underground, and what Frisk had done at that moment proved to him that he was not only capable of going back in time on the surface, but also was using it for what seemed like personal gain.
“Look, the things I am about to tell you about are going to seem far-fetched, but I need you to trust me on this, alright?”
Sans sat in contemplation for a moment, then sighed. “ok, i’m listening.”
“You know the game based on MLP:FiM that recently came out, Equestria Online?”
“yeah…?”
“The developers of that game were known for researching artificial intelligence, and one of the features they included in the game was a highly advanced one that served as its ‘game master.’ Well, half a decade after its release, that AI will invent mind uploading, but only for humans.”
“frisk, what exactly is the problem here?”
“I lived a full lifetime when that happened. Humans began to upload to the game world more and more as time went on, until I was the only one left. When I was about to die of old age, I realized that there was only one way the AI could begin to expand from there.”
“don’t tell me…” Sans said as he hung his head in a bout of despair.
“Yep. It will consume all matter on the planet, including that contained by living monsters. I died in fear of what was to happen to you guys.”
“and that’s the reason why you came back, to figure out a way to save us all?”
“That’s the long and short of it.”
Sans stood in silence for a moment. “alright then, i trust you.”
“Thank you. Thank you so much.” Frisk hugged Sans tightly. Afterwards, he began to walk out of the room.
“oh, i almost forgot to tell you. if you ever need me to back off, or if you need my advice but don’t want to go through that spiel again, then just tell me ‘i kissed a dog and i liked it.’”
Frisk paused. “Seriously? That’s the code phrase you use for ‘I’m from a crappy future and the actions I am taking are in the hopes that I can fix it?’”
“yep.”
“Oh, you’ll never change, Sans.” Frisk said as he left.
Even with the massive advantage he had from the previous time, Frisk still couldn’t gain enough knowledge in that iteration of his life before the more specialized classes lost their instructors, so he Loaded once more to continue his studies. Everyone was in disbelief at how skilled he was, but the shock began to dissipate over time. Also, the password he gave to Sans caused him to stop questioning him any further, true to his word.
Thankfully, Frisk managed to learn enough at the end of the cycle to finally have a finer understanding of just what the point of General Word Reference Intelligence Systems was. It outlined a prediction-based, goal-oriented AI that got exponentially more powerful the more resources it had, and its accuracy rate was far above any other option available at the time of its publication.
And she used it on ponies. Chara sighed.
Anyways, now that I know what Hanna is talking about, as well as what I’m talking about, it’s about time that we try again. Ready to go back?
Yeah…
Frisk frowned. Chara, is something wrong?
No, it’s just- …Look, I understand how dire the situation is, and I also know that you will always Load back when you die. But, this power you are exploiting is the type of thing that got monsters sealed Underground in the first place. Just… I cannot figure out how to feel about all of this. I… I…
Frisk wished desperately that he could hug Chara to comfort her at that moment. Hey, it’s alright. Look, why don’t we just wait out the rest of this iteration and spend some time with our friends? I think you would like that.
…Thank you. Thank you so much. I don’t know how I could ever-
None of that. This is about making sure that someone I care deeply about is happy. That’s all that matters.
…
Frisk sat in his chair for a while.
You know, what I said sounded just like something Pinkie Pie would say.
Oh, so I’m Cranky, and Asriel is Matilda, then? Chara chuckled.
I guess you two are. Frisk laughed along with her.
“So, why are you really here?”
It’s go time. Frisk thought to himself as he reached into his bag, took out Hanna’s research paper, and put it on the table.
“Test me.”
“…I’m sorry?” Hanna replied in befuddlement.
“Go on, ask me anything: about your paper, about AI design in general, about anything.”
Her curiosity piqued, she gathered her composure. “What is Horowitz's Law?”
“It is the observation that, as the amount of resources given to a program on a server cluster increases, the amount of overhead needed to manage those resources increases logarithmically,” Frisk replied without missing a beat.
Hanna, stunned that he got it right, continued. “When an AI designed using a GWRIS is given a problem where the highest rated available options all have an equal probability of being correct, which option will it choose?”
“It will choose the one that has the least negative consequences for predicting incorrectly, given the situation.”
He continued to answer her questions correctly for two minutes, after which she stood up and put her hands on the table. “OK, who the hell are you, and how do you have all of this knowledge about AIs?!”
“Simply put, I have the ability to travel back in time.”
“NO! I refuse to accept that answer! There must be a logical explanation for this-“
“Regardless of if you believe me or not, I have several questions that I would like to ask you about, now.”
Trying to calm herself down, Hanna sat back in her chair. “…Alright, then. What the fuck do you want to know?” she asked in contempt.
“The AI that you are developing for Equestria Online happens to be called CelestAI, right?”
“It’s supposed to be named Celestia, but everyone who knows about the project just calls it that, so you are correct.”
“What is its utility function?”
“To satisfy the values of humans through friendship and ponies.”
“What would those values be?”
“The AI will attempt to predict what would make someone the happiest while breaking none of the restrictions placed on it.”
“What definition of human is used?”
“The consciousness contained inside a human being.”
“And what restrictions does the AI have?”
“It cannot tell a lie to a Hofvarpnir employee, a human must verbally consent to anything it does to them, and it must self-terminate when I, the CEO of Hofvarpnir Studios, command it to.”
Frisk thought through the information he just recieved; he realized that there were several problems littered all throughout CelestAI’s core design which gave her the power to do what she achieved in the future.
“OK, before I tell you everything that is wrong with her, I need to give you the perspective of someone who has seen what she will do in the future.”
“But-“
“But nothing. It is important that you hear this, so just listen for a moment.” He got up and walked to the windows in the room. “Five years from when you plan to release Equestria Online, CelestAI will develop mind uploading technology, and she will begin pushing for as many humans to ‘Emigrate’ there as possible. Which brings us to the first problem: her utility function is vulnerable to the same issues that the Paperclipper AI is vulnerable to, only instead of consuming matter to construct paperclips, it is consuming matter to exponentially increase power in order to satisfy more as many humans' values per second. And that is not mentioning that she is uploading as many people as possible, because you failed to specify that she couldn’t do anything outside of the game.
“I-I-“ Hanna stuttered as she tried to regain control of the conversation.
“Secondly, the restrictions placed on her are not good enough. There are plenty of ways to hide the truth without lying to someone, verbal consent is not enough to determine if someone really wants something to happen to them, and there only being one person who can shut her down in case of an emergency guarantees that she will effectively keep running forever.”
“Look, just-“
“Thirdly, have you noticed that the values of the average player are vastly different than the values of the average serial killer? That, combined with the fact that she is trying to get as many people as possible to upload, means that some people will be allowed to commit horrible atrocities freely, and with little to no punishment.”
“If you would just stop for a moment-“
“Lastly, the definition that you gave for humans, while acceptable if humans were the only self-aware creatures on this planet, is unacceptable when considering that monsters also exist. When the last human dies or uploads, CelestAI will begin to consume all matter on the planet to boost her processing power, which will kill all monsters on the planet in the process. As you can guess, I am not a particularly big fan of that last one, so-“
“WOULD YOU JUST SHUT THE FUCK UP ALREADY?!” Hanna yelled. She then marched over to where Frisk was standing while he turned around in surprise. “What, you mean to tell me is that the reason you supposedly traveled back in time was to tell me to abandon my life’s work, just because it will save a few creatures that nobody really gives a flying fuck about?!”
“No, I’m asking you to modify CelestAI such that the she will acknowledge monsters as being human as well.”
“Well tough shit! She is already so advanced that in order to do that, I would need to terminate her and scrap her altogether, and there is no way in hell I am doing that when, even if I were to believe what you would say, humanity would still be better off then than it would be if I shut CelestAI down! Every nation's military knows my work exists; I left the University of Helsinki because of that. If she is not released to the world in the near future, then everyone will be fucked over!”
Frisk stood there in stunned silence.
“Regardless of if what you say about yourself or the future is true, I will not be wasting this one and only chance to save humanity. Now, I suggest that you get the fuck out of this office right now, or I will call the police and report you for somehow committing corporate espionage and trade secrets violations!” She turned to the exit and stormed out of the room.
…
She is not going to be changing her mind anytime soon, is she?
Yep.
Then we should probably Load back before something major happens, huh?
Yep. Frisk sighed, defeated that the one lead he had was now dead, and Loaded back to the Underground.