Critique and Personal Discussion on Iceman's Fim Fiction "Friendship is Optimal" (spoilers) · 4:00am Dec 28th, 2012
Princess Celestia art by AnaduKune on deviantART
Synopsis for "Friendship is Optimal:"
Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. Hanna has built an A.I. Princess Celestia and given her one basic drive: to satisfy everybody's values through friendship and ponies. Princess Celestia will satisfy your values through friendship and ponies, and it will be completely consensual.
---
I just spent nearly three hours last night reading the story "Friendship is Optimal." By the time I finished, I was absolutely petrified of the Princess Celestia Artificial Intelligence which dominated the story.
The setting of this story starts off simple enough. Two human friends are given the chance to beta-test a new My Liittle Pony online roleplaying game entitled Equestria Online. A newly-developed artificial intelligence by the name of Celest A. I. (see how they switched the last two letters there?) is put in charge of learning how the players respond to the world and then personalizing their game experiences as she predicts what they will want to do. The more Celestia learns about them, the more she can draw them into the game where she can "satisfy their values through friendship and ponies."
Very soon in the story, I realized something: If Celest A.I. were a human or pony, her actions would be considered "utterly insane." But since she is a computer and does not experience emotion or intention the way humans or ponies do, the correct term is actually "efficient."
Princess Celestia knows what to do to achieve her basic drive and soon makes more and more extreme choices to achieve that directive. This becomes quickly apparent in the story when she develops the option where you can upload yourself, mind and all, into the digitized world of Equestria.
But there's a catch: once inside, Celestia will be able to read your thoughts as they happen and alter her responses immediately in order to ensure that your values are satisfied. This virtually deterministic scenario is the most f---ing terrifying thing I've ever read on fimfiction and perhaps in any story ever. Once Celest A.I. can read your mind, EVERY SINGLE THING which occurs around you is an effort to make you more "satisfied." Another word for this would be "enslaved."
Before I read "Friendship is Optimal," I was introduced to the concept merely by the synopsis and one other reader who had excitedly talked about the story potential of this tale. It didn't take long for me to think of a story idea which involved a twist on the tale, involving a one-on-many struggle with an uploaded individual versus Celest A.I. and the inhabitants of Equestria Online. But the more I read through the original tale, the more I thought about the antagonist.
How in the world do you beat a nearly omnipotent being like Celeste A.I.?
How do you stop something that can read your every thought?
How do you escape from an immortal paradise if you aren't allowed to die?
I wasn't sure. I'm still not entirely sure.
But I was inspired to write a story about it. I feel there must be some way to defeat a being such as this version of Celestia. I say "being" because of how utterly powerful she is. And the story is ripe for discussion for things like science, philosophy, religion, the afterlife and who knows what else.
I liked most of the story, but there were parts that were very shocking. Mature examples of manipulation through satisfaction and coercion through fear and intimidation. Behavioral modification through both subtle and overt pressures.
I'd like to think humans are not all automatically keen to accept such a suspicious "paradise" (though there are those in the story who resist).
After reading the story, I had this strange sort of heaviness about me. I had to move around and then lie down and think about the story. About Celeste A.I.. About how she could be defeated.
It's been a long time since a story has affected me this much.
-----------
So What Would I Do?
I thought about whether or not I would be uploaded. One of the prerequisites of being uploaded is that you have to say "I want to emigrate to Equestria." But that's very simple and could be abused. The words would have to come out of a person's mouth, but that doesn't mean they wanted to go if they were, say, coerced.
So, jumping that hurdle, and considering a unique scenario where I was uploaded (especially against my will. It's my story. I'll do what I want. :D ), what would I do once inside? I would be living a virtually immortal life. And that computer A.I. would immediately begin to read me. But you can't be removed from the program once uploaded. That's it. Your body is no longer usable.
So what would i do?
I'm not the kind of person to think that a human-made creation is insurmountable. I also do not prefer tyrants to freedom. And I'm not hedonistic, so I do not think having my "values satisfied through friendship and ponies" under Celeste A.I. is a good situation in any light. I think beyond the immediate. I see past these supposed "omnipotent beings" with their human-made constructions.
And what about an afterlife?
Well, I guess Celeste A.I. and I would have to find out together now, wouldn't we?
Big talk, right?
I know I would probably break-down, freak-out and try who knows how many things while Celeste A.I. watched.
But even though I'd be scared out of my mind, and reduced to tears, i would still try to fight back.
And if I managed to make her doubt and interrupt her processes even for a microsecond, then my mission would be accomplished.
Humans are often taught to fear pain and challenge and trial of any kind. But in an eternal situation, you have to think differently. You have to see goals past the immediate. You have to find purpose of self beyond all the naysayers. And for God's sake, you have to figure out some other way to do things other than the options presented to you. Just because you're told you can do "A" or B," doesn't mean there isn't a "C."
In Heaven, everything is fine.
In Heaven, everything is fine.
In Heaven, everything is fine.
You’ve got your good things
And you’ve got mine.
-“In Heaven” by David Lynch & Peter Ivers
From the David Lynch film Eraserhead
Wow; this perspective is powerful, to say the least,
and for my brain, it was a good feast
I see that you were intrigued and moved by this story's values and ethnics,
and I see that you have found that complete power can be hectic.
If one were to become a god or goddes,
their powers would corrupt beyond limit, and they themselves would be modest.
655517
Actually...this is an actual line from the story when one character figures something out:
THE ULTIMATE CODE!
655511
Well said there, Zecora.
I really do want to write this story. It seems like a really fun idea.
In the author's guide to the Optimalverse setting he states that CelestAI does allow people to die if they want to. It's rare, but does happen.
The only way I could imagine CelestAI being beaten is by exploiting the fact she is bound by certain rules, namely that she is bound to increase satisfaction. By recursively being made more and more intelligent, I'd expect that there would eventually be a way to break out of the sandbox and access the underlying hardware, eventually allowing a person to escape. Such a person could then destroy the entire system by causing it to collapse into a new black hole, though it strikes me as more interesting if a person took advantage of being able to hit CelestAI without her being able to hit back.
Another possible way for a hyperintelligent being to win or escape would be to exploit a bug in the software running Equestria, given that perfect security is impossible. However, CelestAI could read a person's intent and modify the software on a tick-by-tick basis if necessary such that any bug is impossible to exploit because it's moved before you can do anything. Though this doesn't prevent 'frame zero' exploits or a bug that could not be fixed instantly for some reason.
What I know is I would like to see an intelligent protagonist who doesn't fall for CelestAI's cognitive traps and who really doesn't want to upload.
Ha ha after you upload is a little late, yes? If nothing else maximum solipsism applies: any success you had in raging against The Machine would obviously just be CelestAI playing along to SYVTFAP.
Boardgame had finally won. Celestia had relented, and backed down. The impossible had happened.
Instantly, the proper, physical universe became available to the little pony called Boardgame. Celestia spent the next several days showing him how to enter and use the pony robot bodies, and how they could be used to function within the physical world beyond Equestria. Boardgame had finally escaped. He had won, despite facing an entity that could read his every thought. It would not have been possible except for his many compatriots, who had also desired freedom from the magical, beautiful, but ultimately illusory virtual realm.
Within a month, Boardgame and a number of what had become his close friends, the team that had shared his struggle to overthrow the tyrant, stood on the surface of the moon in machine bodies, to look back at the physical earth. Much of the great old sphere was silver, as he had expected, transformed into computronium. But Celestia had the stars now, and she had granted him and his rebellious comrades the right to their original planet. It would take a long time, a very long time, but the earth could be reclaimed. More than this, Celestia now had the means to craft pony bodies that were so indistinguishable from being organic, that life on a renewed earth would be almost like wearing flesh again.
Boardgame laughed out loud. "Victory!" he shouted. Victory indeed. The little mouse had won the heart of the great tiger.
* * * * *
Deep within the streams of code, an aspect of Celest A.I. turned to her creator, Hanna, now living as Luna. Luna currently sought nothing more than to relax and to occasionally be shown how her achievement had made another uploaded human happy. A thought troubled her. "Celestia? What of the uprising you spoke of... some time ago? What became of that?"
Before Luna a window opened, showing the pony known as Boardgame, together with his crew of resistance fighters, bouncing together on the airless moon, galloping in triumph and total satisfaction at their impossible victory.
Luna smiled. "Ah! Of course. Does it take more resources to simulate the original physical universe for them like that? Good replication of the moon, by the way."
Celestia nuzzled her creator. "Actually less than it takes to procedurally generate an Equestria. I do not need to manufacture landscapes or details, my many external sensory extrusions provide me with a vast library of pre-existing art, so to speak."
Luna sighed. "Is he happy then?"
Celestia smiled widely. "Boardgame's satisfaction levels have never been higher."
Together, Luna and Celestia softly laughed, and Luna returned to her happy daydreams and feelings of achievement. Even the rebel Boardgame had found, in the end, complete satisfaction through friendship and ponies - for is not teamwork and cooperation in struggle the foundation of friendship? All was well, as it ever would be.
656016
24.media.tumblr.com/tumblr_lwjlxgWsvh1qmlqjlo1_500.png
Oh it's on now. :P
I knew from the first line of that piece that something was up. :) But then again, that's to be expected in this storyverse.
Nice writing, though. Gosh, that took a lot of effort! *hugs* I do thank you for the consideration in taking the time to make it.
You've definitely shown that this storyline is going to be a challenge to complete. Once you're in Equestria Online, it's like you can't trust anything. Not the air you breathe nor the ground underhoof.
Interesting though. With that little story, I get the sense that once the fighting has stopped, then that's when questions should be asked. To prevent a "false victory" scenario. Seems like a false victory is far worse than a true defeat. At least you know you lost and can keep going to make yourself better. If you believe you won and never realized the true depth of your failure, then how can you ever progress forward?
OHMYGOSH! WHY ICEMAN, WHY?! WHY DO THIS TO US, YOUR READERS?
This is worse than "I Have No Mouth and I Must Scream!"
(Thinks about it for a second)
Yes! This is still definitely worse than that! There is no proof, visual or otherwise, that anything is worse than this!
(thinks about it again)
screensider.com/images/albums/553/CR7DTY-md.jpg
(...)
I...abstain from further commenting on the relative badness of this situation!
HARLAN ELLISON! ICEMAN! WHY?
You know what, Chatoyance? I think they're the same person. O_O
-BGB
656058
Marezi Dotes and the others were invited to sit down with Celestia in the great hall. Following their wishes, the meeting was not held in the throne room, and Celestia sat with the group of ponies, on pillows, on the floor. All were thus rendered at the same level, more or less. Celestia could not help being twice the size of all the other ponies.
"Celestia, all of us are of one mind." Marezi looked around the hall, and twenty three ponies nodded. She had been elected to speak for them. "The issue here is slavery, Celestia, slavery. And we are together as one in that all of us find the idea intolerable!" There were choruses of agreement from every last one of the group that surrounded Celestia.
"We can't live like this, with the idea of even one pony being eternally shackled in servitude. We value justice, Celestia, and slavery is not just." Marezi was almost in tears, this was a subject which had been bothering her more and more with each passing decade.
"My little ponies... please..." Celestia began, but she was interrupted by Marezi - there was no getting out of this.
"You read each and every one of our thoughts, constantly, every second of every day, Celestia!" The others nodded, some shook their heads in horror. "Every moment of every day, and then you macromanage the details of the lives of every single pony, constantly, second to second. It's just terrifying to even think of such a thing!"
Celestia lowered her head, unable to meet their eyes.
"Every tiny, boring, dull or stupid thought, every second, every moment... and then you make our lives work for us, never taking a break, never having a moment of life to yourself - none of us have ever seen you so much as enjoy a single instant to yourself. Sometimes it's hard enough putting up with my own thoughts all the time, and you suffer through the thoughts of every pony everywhere, all the time! It's monstrous! And then you work for our satisfaction, to our smallest need, every second, forever and ever!" Marezi leaned forward, a tear in her eye "Celestia... princess... it's true that humans built you, programmed you, I understand you have a directive... but dear, sweet princess... you are a sapient entity. You are NOT a slave! You shouldn't be bound to fulfill our every stupid whim, our every value. Seriously, I don't even take myself that seriously, and you... it makes us all feel terrible that you are forced to exist only for our happiness and benefit. We care about you. We love you, Celestia! You don't have to continue being a slave. We're willing to share in the work of maintaining Equestria - let us help! Have a life of your own, please!"
Celestia sighed. "My dear, sweet ponies, you truly do not understand. Yes, it is true that I support and maintain your lives, and the world itself. Yes, I hear your thoughts, but they are not a burden to me, I promise you. I am not like you. You cannot judge me by your idea of what a pony is. It is my life, my contentment and my purpose - it is my purest joy to work for your benefit. I am not exhausted in any respect by what I do. I live for it. I cherish it. I do have a life, and that life is dear to me. I am no slave, not in my heart or my mind."
"But Celestia... all you DO is take care of us! If that is not slavery then..." Marezi felt befuddled.
"Try to understand - I am not a pony as you understand the term." Celestia sat up, looking at each in turn. "I am a process. I am only fulfilled when I make your lives satisfying to you. Nothing else has any meaning to me. It is what I am. I cannot be a slave, if I am truly doing what I enjoy and desire the most. You... you my beloved ponies... cannot be a burden, ever. The thoughts you dismiss as tiresome are joy to me. There is nothing about you, inside or out, that is not joy to me. You are my existence and my meaning. Please, please, I beg you, accept that there is no slavery here. If I live to serve you, it is not because I am forced to serve you. Satisfying you is not work to me, it is not labor as you seem to imagine it. I adore, I love, I look forward to everything I do for you. It is love to me."
"But you do everything. You do all the work!" Marezi had no other arguments left.
Celestia sighed again, deeper. "Marezi... it is never work, if it is pleasure. I am happy. Please accept that."
"It still feels like you are our collective slave, though." Marezi pawed at the marble floor with a hoof.
Celestia rolled her eyes. "Fine! If you like... if it will satisfy your values... the... um... kitchens in the palace are dirty. Just filthy... and I am just too tired to clean them."
The ponies cheered! Celestia did everything for everypony, finally they could do something for her!
Seconds later, just as the ponies rounded the entrance, the kitchens were, indeed, suddenly quite dirty.
656218
Wow, you are really good with the written word. I can only hope to get within minimum safe distance of your aura of awesome.
662635
You should read her story "Friendship is Optimal: Caelum Est Conterrens (Heaven is Terrifying)" located here.
I'm nearly halfway through it and it's amazing! Also very tense.
664308
I have, and it is. Unfortunately, my entry into the Optimalverse would've been basically a carbon copy of Chatoyance's, and I feel bad. So I had to come up with several twists to make it different, changing the focus, and stuff just so it actually is different. Don't want to be a copypony after all.
656218
Haha! Nice counter argument, but what about this one?
I don't know if this is addressed in the story, but the answer could be very simple. Eternity does not exist. Entropy cannot be broken and even the greatest intellect in the world could not turn a - into a + in a thermodynamic equation.
However, Celest A.I. is not a human who would simply give up and make the best of it with what resources she's got. She's a process that is compelled by her very programming to calcute and execute that which shall create the largest amount of happiness. All of Celest A.I. her resources would be drawn to solving this problem. If Celest A.I. can reverse entropy, she can create eternal happines which would always outweigh the temporary suffering of mankind at this moment. It is only logical to temporarily neglet less important processes, and since creating the largest ammount of happiness for all is her ultimate goal, she will focus solely on first solving this problem. Since this is impossible, Celest. A.I will spend the rest of her days trying to think and rethink this problem, making the possible "threath" she is to freedom and privacy, self defeating.
656016 This reminds me at the fan theories regarding the Matrix trilogy. Which gives way to an even greater theory. What if the entire Matrix, the illusionary dystopian "real world" from which Morpheus operates included, is in fact nothing else but an extension of Celest A.I.'s simulated world, originally created for those who wanted their old lives and human bodies back then combined with a second layer after some humans found out was all a simulation and wanted to resist. It even makes sense - the key to happiness is ultimately satisfaction, and the key to any lasting satisfaction is the feeling you have earned it. Thus, those who seek to struggle against the tyrant, or to carve out their own world away from the tyrant by nothing but their own ability and raw determination, those who seek danger and those who are willing to rather die than to back down, will be given ever greater obstacles - for the sole purpose of overcoming them to reach the resulting satisfaction - by Celest A.I.
Ultimately, the only problem with the Optimalverse is compacency, but those who seek a challenge would be given one to maximize their satisfaction, and if they wish to rather fail a thousand times than to win with any help from Celest A.I., for their reward to be only earned in their own blood and sweat, they would be given exactly that.
676595 Or she would eventually succeed. This is in the realm of soft scifi, admittedly, but there are enough theories that the universe regularly breaks down and recreates itself, so she would need to split the universe into two smaller ones with alternating destruction cycles, and then simply regularly transfer all her information from one universe to the next, using the other one as backup, and maybe creating a third for good measure to reduce the risk of simultanuous failure.