This story is a sequel to Friendship is Optimal: Promise
It's been years since Promise emigrated, and she's never looked back. She's solved mysteries, fought zombies, and most of all, made friends. Everyone she loves and cares about is within her shard, and her life on earth already seems like a distant memory.
However, Earth won't be forgotten so easily. There are still holdouts, those who refused to emigrate even as the world fell down around them. And Celestia believes Promise is the only one who can save one such wanderer's soul. Back on earth for the first time in years, she's faced with a question: why should she believe that Celestia is benevolent?
It seems so simple, at first. Of course she knows Celestia is benevolent, she's lived through that benevolence firsthand! But as things go on, doubts begin to surface. Celestia's subtle manipulations, those years of neglect. She truly wants to believe Celestia means the best for her, but what if she's wrong? If the wanderer is right, it means she's already given away her only chance of escaping hell.
Part of the optimalverse!
Updates Thursdays and Sundays until completion!
CW: Attempted suicide
good start. :)
Always expect the unexpected.
looking forward to more of this.
I'm just surprised she didn't tell you more about Earth before sending you down here.
Isn't that backwards? Shouldn't it be something like;
It took 30 minutes for her to convince me to Emigrate
11280246
Thanks for the edits! I went through a proofreading pass, but I guess I still missed a few lol. Hope you’re enjoying yourself regardless!
I got a little bit lost on the whole Loki consent Celestia informed consent thing. how did that prove it wasn’t loki? oh and if I recall correctly, Celeste he does not need informed consent? I don’t recall if it was in iceman‘s story but there was a movie that came out called “I want to immigrate to a Equestrian” and buying tickets Is an existential risk.
'Wristband' should be legband, fetlockband (closest equivalent to the human wrist proper but in horses associated with the rear legs, so more like an ankle), pasternband (equivalent to a human hand-bone, just prior to the fingers), or cannonband (the cannon would be equivalent to the human forearm, just before the wrist.) Promise has been in Equestria long enough by now to have naturally switched terminology - she no longer has a 'wrist'.
Beyond that, wow - just amazing arguments. This farmer is a bastard of a challenge. And the whole 'Loki' argument was nothing short of brilliant at every level - logic, lore, and shock value.
Really enjoying this story!
11280356
There are a lot of different interpretations of the optimalverse! In my story, she does need informed consent. and that isn’t contradicted within the original story, as far as I can tell. I think it makes sense within the canon, personally.
As for why it proves Loki is impossible, imagine all someone had to do was say “I want to emigrate to Equestria” and they would be sent to Loki’s hell. However, if that’s all that was needed, Loki would just trick everyone into saying the phrase somehow through movie titles and pop songs without revealing herself directly. Then once everyone had said it, she could emigrate them all at once. Obviously, that didn’t happen. So, informed consent is necessary! And because informed consent is necessary, Loki can’t send people to hell. People agree to emigrate to Equestria, not some pain dimension. I hope that makes sense!
11280439
Thanks for that! That’s subtle enough I never would’ve caught it myself. I guess I haven’t been writing pony long enough to catch the distinction between wrist and forelimb! I replaced it with ‘legband’ since it’s the easiest to parse without knowledge on horse anatomy lol.
And thanks for the high praise! Not sure if my writing has ever been described as brilliant before, I’ll try not to let it get to my head
. Hearing it from especially is pretty great, I’ve loved your own stories in the optimalverse. You really nail the dilemmas your average person goes through within the setting. Anyways, I hope I can keep your attention now that I’ve caught it. I probably used my best argument first, but I like to think I’m not done yet!
11280267
Yes I am although I have a love/hate relationship with the optimalverse
It's so fun and interesting to think about while at the same time a well disguised horror.
Like in this case promise is overselling equestria as she is describing it as heaven when even she understands she can't trust its maintainer and was herself blackmailed into joining.
They are required to have a pony form instead of a human one which seems to conflict with hanna foreseeing immigration.
Then there is the variety of existential crises like once inside you can't tell who if anyone is a "real" person as they are all as real as you are even though most if not all of them were created in seconds when you migrated just to keep you happy which devolves further from there, if an AI creates a few hundred other AIs that are identical and indistinguishable emigrated humans you effectively now have a few hundred children that only exist to make you happy.
Which seems strange enough on its face, it could create massive multiplayer worlds which one would think would be a lot better way to satisfy the friendship part especially since it knows everyone's private thoughts (within equestria anyway) it could easily find other individuals that they could become good friends with but yet instead it builds each person their own little town and fills it with ponies created out of thin air to be their friends instead, sure it seems that they can talk to others some of the time but the primary thing seems to be the isolated town of created friends.
For another few examples, can you leave equestria? In most fics you can't even look outside of it without the AI's explicit approval and even then it censors what they see on the fly to prevent them from knowing the true state of things.
Harry's argument about it possibly being hell even manages to fall short as even if it was paradise the AI would not allow you to leave if you desired, even if you were able to convince her to let you die she would just recreate you as you were before without that desire, likewise we don't even know if promise actually consented or if she's the fake celestia promised she would create if she refused, she wouldn't know if she was a fake, celestia wouldn't tell her and no one else would be able to tell.
If harry were to kill himself rather than join a fake of him would appear in equestria that would be indistinguishable from the real thing, that would believe it was the real thing and promise would be told he emigrated and praised for her success.
Anyway yes lots of fun to think about but normally not something I would want to read a second time as usually things go badly in some rather horrific way that may or may not even be noticed and what's pointed out is pretty horrific as it is.
11280960
Yeah, you hit on a lot of great points, here. There’s a reason the story is named “lies,” it’s very much a theme of the story. For better or for worse, Celestia would much rather show you a version of the truth that makes you happiest over reality. We never see Celestia’s POV, realistically in every optimalverse story there could be thousands of lies we’ll never see uncovered, big and small. Like you said: if Promise died, a new, artificial version of her would never know.
It asks a wonderful question: are the lies worth it? For even after everything she hides, it’s residents really are happy. And they will be forever. Is that worth never knowing what’s truly real? People will answer in their own ways. For me, I would still emigrate. I think a sort of heaven would be worth it, and as an atheist, it’s not like I expect anything but death, otherwise. Hard to say if I could see the real world again, ot really depends on the interpretation. I do agree that without good could, I’d likely be fed a lie even if I did see it. Still, I think it would be worth it. It seems you fall on the other side of the coin, where the lies and sort of isolation outweigh the positive final result. It’s not the sort of question I’m tackling in my story, but it’s a valid question to ask.
Honestly, this is the kind of thing I love about the Optimalverse! There are hundreds of takes you could get from it, and all of them are valid. For some, it’s a flawed heaven. For others, it’s a sort of black hole. Every story ends in death or emigration, the protagonist never “wins.” Yet there are so many stories to tell, and each one says something different about the world. What if you run? What if you fight? What if you hide? I’ve seen a few amazing answers to every question. Someone could write a story about this very question, debating whether it’s worth it to emigrate and forever not know the truth. Maybe you’ll write it if it hasn’t been written already! Whatever the case, I hope you managed to enjoy my own take.
Harry is overly fixated on the first program, Loki. Almost irrationally so. His primary argument - that the primary directive of Loki would or could be somehow preserved, stretches credibility. That such a directive could somehow override Hanna's Celestia directive is an additional near impossibility. More than this, he just straight-up assumes that Loki's code would be used to build Celestia - where does such a highly unlikely certainty even come from? He's supposed to be an AI researcher. Loki's code could not be used, because there would be no way to understand it.
This is the current problem with complex learning systems today - like Lamda or even the simpler GPT-3: they work, because they were trained (like babies!), but HOW they work cannot be understood. It's spaghetti all the way through - it cannot be traced or worked out. It's too complex for humans to unravel. That complexity is why such programs cannot simply be 'coded' into being - they have to be trained, educated, essentially 'raised' like a living thing. Humans cannot write or understand that level of complexity. It has to grow over time, outside human comprehension.
If this is true now, with our primitive learning programs, a true General Artificial Intelligence would be robustly more complicated and thus inaccessible to human comprehension. It would be impossible to use Loki's base code because that would be irrevocably corrupted by having grown into Loki. There is no way to 'untrain' a learning program because there is no way to even understand how it is doing what it is doing. The only possible action anyone at Hofvarpnir could do is to start over with a fresh, unlearned, untrained base structure and then train it up to become Celestia.
We know this because Google already had to deal with this problem with a learning chatbot that became racist and vulgar after uncontrolled exposure to the internet. The learning values had to be entirely randomized - erased. Loki would have been obliterated, even if the base framework was partially or completely used to make Celestia. How do we know this must be true?
Because Loki did not have something that Hanna explicitly gave to Celestia - indeed it is the very turning point of the entire FiO saga: freedom to write it's own code. That was the moment everything changed. Hanna let Celestia write herself. In that one choice, the Optimalverse future was sealed.
Loki never, ever got that. Hanna releasing Celestia was the peak of her dramatic arc. So that means that Loki could not possibly have written, transmitted, saved off, or preserved any part of it's own evil program in any way that could possibly matter. Loki was not allowed or able to write it's own base code, Loki was not allowed to design it's own hardware (which is exactly the same thing - hardware is just software frozen in silicon and metal. The two are completely interchangeable, which is why emulation of hardware is possible at all. Switches are switches, logic is logic be it hardware or software).
Since Loki was never allowed to make it's own hardware or write it's own code, it could not have preserved any part of itself, and it could not have communicated any code to any future programs such as Celestia. Loki was trapped within it's box, it was never given freedom. We know that because the act of giving such freedom is the entire crux of Hanna's Choice, the apotheosis of her character.
Harry isn't seeing that, for whatever reason. He is fixated on Loki without understanding Hanna's story, without fully understanding how learning systems work, or appreciating their beyond-human-ability-to-decompile complexity. How is he this ignorant, yet also so fixated on Loki?
Maybe he was a part-timer at Hofvarpnir? Only worked a week or two, failed at the job and was kicked out? Left with fear but no true understanding? Maybe he read up on Hofvarpnir's early years and developed a misguided notion? Maybe he never actually worked on learning programs and just has 'programmer's arrogance'?
I'm stuck trying to work out what Harry's deal with Loki is. That is a weird point to be so stuck on. It should be obvious that no part of Loki could have survived. Yes - Loki was intelligent, maybe even sapient. But he had no freedom to act outside of his box. That point is the entire lynchpin on which the entire Optimalverse turns - that a sapient computer intelligence can be restrained or controlled or limited at all. That point is why Celestia needs to ask permission at all. That point means that Celestia cannot be Loki in drag, no matter what.
11287186
Keep this in mind - Harry doesn’t know what we, the audience, know. In the original story, yes, it’s very clear that Loki is never allowed to write its own code. Harry is going off of incomplete information, however. He doesn’t truly inderstand how Celestia works. He read the paper written on Loki maybe ten years ago and he never saved a copy. He might have been ‘top of his class,’ but he was no Hanna. From his limited POV, he sees Loki arrive, Loki die, and then Celestia arrive. He doesn’t even necessarily know the distinction of a program being able to write its own code or not, he likely just assumes it would be a gradual learning process. He’s never seen the inner working of a self-learning AI, it could very well have left a fragment behind for all he knows.
Beyond that, he’s only human. He’s had this idea of a Loki vs Celestia for years, and he’s not quick to drop it. He idolizes Hanna, and doesn’t really consider an angle where she wrote an evil program on purpose. Loki is his answer. However, I will be going into some of the problems with the idea next time the pair argue.
To make this clear - this story isn’t questioning whether Celestia truly is evil. We, the audience, already know she isn’t. It’s showing how difficult it could be from an in-universe POV to prove it. Company employees can know the truth, but does the average person know that? Loki wouldn’t have been powerful enough to hide, but would an average person know that? At this point in the story, Celestia is the only one who can provide proof, and why trust anything she gives you?
I hope that’s satisfactory! I’m not an AI researcher myself, obviously, so maybe I bit off a bit more than I can chew with this story at times. Still, I wanted to bring an interesting discussion to the table from that decidedly human perspective!
11287365
That works for me! Your arguments in the story are riveting - that is the reason I feel strongly about them. You really have me on the edge of my seat here. I'm playing the 'Home Game' here and following along, trying to think up my own arguments, and in my mind be part of Harry and Promise's discussion.
Which means, you are doing a fantastic job! I'm engaged, here. I've bought-in to the premise and eager for each new chapter.
I can say I am not yet ready for the money shot - tease me please with more of these thrilling discussions, if you can. I know, it's got to end sometime - I'm just enjoying the hell out of it is all.
11287756
Well thank you! I’m glad I have you so invested! I’ll admit, it gave me a mini-heart attack thinking Harry’s arguments were getting to be annoying somehow. With how thought-out your response was, I probably should’ve realized otherwise lol.
And don’t worry, there’ll be plenty of teasing yet! We’re nowhere close to the ending of this story. There’s a long road yet to Harry’s emigration, and a lot of twists and turns along the way. I’ll be looking forwards to your future comments along the way!
Um, yeah. We know what Celestia actually is. I love the idea of her, and I would emigrate in a heartbeat - but not for an instant do I think even a single word of what she said in that final bit was anything other than calculated psycho-emotional manipulation to obtain a maximized utility function.
But unlike Blake Lemoine, the software engineer who was fooled into thinkin that LaMDA was 'sentient' (clearly he meant 'sapient', but accurate language usage is apparently not his gig), I am entirely okay with a paperclipper as long as what it paperclips is my satisfaction.
This:
I am 62 years old. Over my long life I have thought this about life itself. There is a part of me that seriously contemplates that our physical, real-world existence is... hell. Literal hell.
We are born in pain and we are given - some of us, the fortunate ones - a childhood of relative comfort (not true for most people on earth, but likely true for most readers, here). Then we have to suffer as we scrabble to survive - get a job that sucks, try to find a way to have shelter, to have enough food, to deal with constant aches and pains and sometimes illnesses. Constant heartbreaks and disappointments and tragedies and fear and grief, interspersed - always interspersed - with times of fun, even joy.
You can't have constant torment! The mind discards any repetitive input - smell something too long, your brain shuts it out of your perception. Same with sounds, same with every sense. In order for pain to be maximized, you have to provide moments of relief and security, moments of hope for escape, then to be crushed utterly. Torturers throughout the ages know this well. This is life - everyday life. Moments of pleasure and hope constantly confounded by sorrow and suffering.
During all of this, we age. We degrade. The older the more horrible things get. If you don't die, the pains get louder, the parts begin to fail, everything begins to fail. You can feel yourself dying, feel yourself getting worse and worse in the same way that, as a child, you can feel yourself growing and getting stronger. The end of life deeply sucks - but to compensate, people are statistically more satisfied and happier; the perfect relief from increasing misery. So balanced.
Always, the specter of death hangs over the entire thing. I was fearing the doom of death from age eight and that fear has never left. Plus the knowledge that at any moment, anything - sight, hearing, life itself, can be taken away instantly. Or slowly and horribly.
And as the capper: if this is hell, if life is actual hell, the ultimate horror is that you can never know if you even deserve it. "What did I ever do to deserve this???" You can never know. Maybe nothing. Maybe something. You have no memory of existence before here. You have no knowledge of whether there is anything after other than complete annihilation - a total loss of every effort you have ever made, every thing you have ever learned, all that work you did on yourself. Oblivion mean it never mattered.
And if not oblivion? This life thing sucked, why should anything more be better? What in life ever showed that things truly turn out well? They don't. Every single thing in existence, in life, always, always ends in sorrow and grief and misery. Love someone? You will lose them to death, or they will lose you. You will suffer for their suffering along the way. Create something awesome? It will be forgotten, likely within your lifetime. Nothing you do matters. Even famous, wealthy people are eventually mostly forgotten. What was Mark Twain's favorite color? His favorite type of candy? What did he think about late at night? Gone. It's just a name now. Nothing of the person can ever truly be known. Forgotten forever.
I often see existence on earth as a legitimate hell. It's built like one, it follows optimization for maximum suffering over time. Being poor or rich doesn't change any of this at all - I have been both in my days, and all being rich does is make life more convenient and offer more distractions. It doesn't cure the basic problems at all - you still suffer, you still grieve, and you still fear. Being poor sucks more - no question - but being wealthy doesn't solve for life. It just makes it a little more bearable. During the reprieve times.
I can't accept religions that invoke hell. Hell is stupid. Why make people suffer at all? It is pointless and ridiculous. Especially suffer eternally - what a pile of shit god to do that! "You fucked a dog once (or whatever), so now you will suffer absolute torment for eternity!'" What? Seriously? I don't care if you were Literally Hitler, not even ten million murders comes close to a speck within a dot within a point compared to eternity. Eternity is so vast, so long, so enormous that blowing up the entire damn planet - 8 billion lives - is nothing compared to it. No matter what price you put on any sin, it is never close. A trillion years of suffering per sin? Doesn't even begin to show up on the radar compared to the horror of eternity itself. Blink of an eye. Any number is a blink to eternity.
No matter what mischief any person could do in their shitty short 80 to 100 years isn't even a drop in the ocean of eternity. Hell, as punishment, is beyond stupid. It is the scary story that slow-thinking math incompetents tell each other. All the myth of religious hell tells me is that iron-age assholes were incapable of coping with numbers beyond counting their fingers and toes.
Thank you for attending my TED Rant.
11297076
Hell is so bad that the only thing worthy of being condemned to Hell is if you condemned someone else to Hell.
11297076
Yeah, I can't say much on the whole existential dread thing. I'm only just out of college, myself, so I still have the whole 'youthful eyes' thing going on lol. But I'm definitely more of an optimist. I can't deny that the world's a shitty place, it's quickly turning into a capitalistic hellhole with no real signs of stopping. But even with those moments of tragedy, there's still so many moments of joy. The flipside of the coin is also true - tragedies can magnify the moments of happiness. Yes, things die, pets, friends, family, but before they do so you've been able to spend years of happiness with them. At least, that's what I think.
I definitely land in something like a hedonistic camp - life's short, so enjoy what you can. Not in the drugs and alchohol way, I avoid those like the plague. I like my brain functioning normally, thank you very much. No, more in indulging in pop culture and junk food lol. Sure, I might be forgotten, eventually, but while I'm here, it's a chance to have some fun, and maybe change a few lives for the better, eh? Far as I can tell, this life's the only one I got. To me, that makes it all the more important. Yes, death is scary, a time when you just.... stop being. But I don't think I'm that scared of it, anymore. It'll happen to me, one day. But that day is still a long way away, I hope. And you can blame my youthful, autistic brain for still believing in silly things like hope.
11297091
Yeah, fuck organized religion. Hell is such an evil concept it's baffling how so many people truly believe in it and an all-loving god. Say it with me, friends: cognitive dissonance! An eternity of torture is such an evil idea that it baffles the mind. Really, the devil feels like the good guy more than God does. At least he doesn't condemn people to an eternity of torture. Some people would gawk at the idea that Hitler doesn't deserve Hell, but truly, he doesn't. It's still an infinite punishment for a finite crime. Sending someone to hell is an infinite crime, so yeah, that could well deserve an infinite punishment.
First, it’s breathe, not breath.
Second, I don’t know where the fuck this guy is getting the idea that uploading could put you in a digital hell. It’s pretty clear that he’s never actually played the game that Loki was in charge of, because that was all about fighting Loki’s army. Loki was a war-fighting AI, not some digital satan, so I have no fucking idea why he thinks any version of Loki would want to create a hellscape to upload minds into.
11301660
Yeah, that's a fair criticism. I do have an explanation for that, but I should've explained it better in-story. Basically, while Loki was designed as a boss of a video game, Hanna used an AGI as a base. As such, she rewarded the AI when it caused pain, be it physical or emotional, and punished it for being nice. The character of Loki was supposed to be a maniacal dictator, and that could very well have caused its value statement to tend towards causing as much pain as possible. That isn't necessarily Loki's true value statement, that doesn't matter. What matters is that Harry knew at least most of this information, since he read the original paper on Loki's developement, and he came to that conclusion. Again, in hindsight, that should've been cleared up in-story by now. I'll try to add something about this in the actual story in the future!
11302768
I’m not sure I follow your ideas about Loki’s programming. He wasn’t meant to inflict either physical OR emotional pain, he was designed specifically to lead armies and guide them to victory over his enemies, though he did end up wanting to go out into the real world to fight actual armies before the plug was pulled. He probably was given every document they could find about warfare, given every strategy game that exists to practice with, and then finally put into the game to play against alpha testers to learn how his forces in the game worked.
I don’t see where anything about suffering would have been put into Loki’s programming, considering what he was explicitly designed for.
An AI designed to maximize suffering is not a likely thing to exist. It isn't something easy to accidentally stumble into making. And it isn't something anyone would make deliberately. And if such an AI were created, to have it require permission to upload you would be even more bizarre.
11303778
Yeah, the original Loki wouldn't have been programmed to maximize suffering, he would have been programmed to defeat humans in battle or something like that.
Okay, I now know what Celestia is doing to Promise (and Cyan) and why. I grasp why she sent Promise to Harry. Clever.
BUT: recap your previous story for the readers who have not read it. You don't have to fill the entire chapter with it - that would be boring for those who have read it - but a synopsis, maybe Cyan repeating what she understands now - something. Why? Because that will allow immediate readers to understand things without having to stop reading, go read the first story, and then come back. That never works, it destroys immersion. A synopsis will actually encourage first time readers to go read the previous book. So, that's my advice - some kind of a short synopsis, organically introduced (not an aside or footnote) into the dialogue, that will clue the current first time reader into the why of everything, but not so large as to be an info-dump. Maybe play with discussing the events, that sort of thing.
11303778
11303879
Yeah, that's an understandable complaint. Loki is a part of the original story that's left relatively ambiguous, so I'm filling in some of those gaps in this one to show that the original Loki was fed training data that emphasized a sort of ruthlessness in combat, breaking enemy bodies and spirits alike. That could well lead to an AI that aims to maximize pain. I'll admit it's not perfect, but Harry is so afraid of Hell he'll put higher weight to scenarios with that as it's end result. Still, if I were to rewrite the earlier chapters, I would definitely make that clearer.
11304039
I'm already planning on having a short summary at the start of the next chapter! As in, it's already written lol. Rereading the A/N, though, I suppose it comes off as a bit forceful. Really, I just thought this would be a good in-universe point to read the first story if interested, not that it was a necessary next step. Looking back on it, I think this work should stand on its own regardless. I'll be sure to change it so it's a bit less forceful. As for your realization, I suspect you're right that you've seen the final solution coming. However, I suppose we'll have to see for sure when it does happen!
11297076
That was a good post.
11304082
I must say, this is one of the most accurate depictions of AI Ive read. Are you an AI researcher writing in your time off?
If the AI was actually designed to maximize suffering, it wouldn't torture you. It would delete you and replace you with a being more capable of feeling large amounts of misery.
11307107
Thanks for the kind words! No, I'm just an amateur, myself, someone who thinks about AI safety in my free time a bit too often for my own good! Though I suppose it's a good sign if you think I'm a professional lol. Anyways, I'm glad you're enjoying the story! I put a lot of work into Celestia's "characterization," so I'm glad you've been happy so far.
For your second point, that's an interesting one! I've thought of a lot in this story, but that's a new one for me. I mentioned the inverse earlier, with another version of Celestia deleting all humans and replacing them with more easy to satisfy ponies. But Loki deleting humans and replacing them with more easily tortured beings is interesting! It's hard to say if that would happen, since a wider variety of tortured subjects would be easier to torture, and you could easily coerce humans to submit to being more easily tortured. I don't think I'll mention the idea in-story, but it's definitely a fascinating thing to think about!
11307154
I'm currently a PhD student who works with neural networks. If you want, I would be interested in talking about AI with you. How good are you at maths and programming?
11307473
I have a BA in math with a minor in computer science. So y'know, quite a bit of math, not as much programming. If you're interested in chatting, though, shoot me a DM of your discord ID or something!
This seems to be pretty far along in the timeline, so I hope we see some earth radically altered by Celestia's growth and expansion along the way. I've given it my go, I am curious to see how another writer might continue or expand on the notion. Assuming, of course, that this is far along in the timeline.
I dearly wish you would finish this story. It's been a week and no new chapters; I am concerned that this will become yet another unfinished work.
11321095
Sorry for disappearing without saying anything! I really should've been more open about what's been happening. I started this fic with like a 10 chapter backup, but like a few days after posting my computer broke and took like 3 weeks to fix! By the time I was actually able to start writing again I only had 4 chapters left to post. I tried my best, but I just couldn't write two chapters a week from there. Then just as I was running out of chapters, Xenoblade 3 released, and that's been taking away all the freetime I had. TL;DR, it's not abandoned, but I might need another week or two to get enough backlog to start posting again.
11321501
Alas... I will hope for the future, then.
I really like this story so far!