Over Riding Jeans by Chatoyance
Over Riding Jeans
By Chatoyance
Before she emigrated, Blaise had been very excited at the thought of a truly benevolent general artificial intelligence running amok. That such a thing would happen was inevitable, she would often claim - to her dwindling number of friends who would listen - so the real issue was never keeping that particular genii in the bottle. Rather, the real issue was the nature of the genii that would surely come. Benevolent was the way to go, obviously.
One day, humanity would bow down to its machine overlord. The best that could be hoped for would be that the overlord would be sweet.
Celestia was.
Blaise spent only two weeks playing with her Ponypad before she marched into the nearest Equestria Experience and got herself uploaded to a virtual life. By that time, she only had one friend left, the others all having either emigrated or turned on her for her evangelizing about the glories of cybernetic existence. Randal tried to talk her out of emigrating. His angle was the usual - uploading was death, it was suicide, it was getting your brains scooped out.
"Sorry, but you are wrong." Blaise was very sure of herself. She knew she was smart. She knew what she knew. "It seems like death, sure. Of course it seems like death! Everything in our evolution, everything in our genetic programming tells us the loss of our body is to be feared! But we are better than that, aren't we? Isn't that the big claim of what it means to be human - that we can override our genetic programing, defer pleasure, accept pain, and make choices beyond what evolution has prepared us to do? More than the sum of our parts, boy!"
"But... Blaise - you are entrusting your existence entirely to a robot! One glitch, one little error and..." Randal was beginning to realize that nothing he said would make a speck of difference.
"Randal. [Randal... Of course that is what I am doing!" Blaise had continued closing down her accounts and affairs while she talked. "Humans make mistakes, they can betray you... but Celestia is beyond that. She is self-repairing, self-modifying, self-evolving! She can't have any real glitch or error, because she can fix herself. She can also work around any problem that might come up. Nothing humans have ever built could be safer!"
"What if she turns on you?"
Blaise shook her head at her poor, simple friend. "Can't happen. Everything she is, everything that defines her is a single, simple rule: she has to satisfy human values through friendship and ponies. That directive is coded into everything she is. It literally IS her. It cannot be denied, ignored, or altered. She is forced to write that rule into everything she does, every change she makes to herself, every new part she adds, however small, however large. It's fractal - the programmer, Hanna, made it so that rule exists at every possible level!"
Randal looked doubtful.
"Listen... Celestia's prime directive is... it's like her DNA. It's part of every bit of her. She can never, ever, ever be anything else but what she was designed to be, no matter what happens. Bye!"
And with that, the life of Riding Jeans began.
For almost three hundred years, Riding had enjoyed the life of a western rodeo pony. Her Celestia had placed her in a shard where she could be with other uploaded former humans that had a thing for the Old West. Her Appleoosa was a shit-kicking, salt-licking, late-night barn dancing western paradise.
Over the centuries, Riding Jeans had been a rodeo queen, a train robber (it was just a game, nopony got hurt), stopped stampedes, roped other ponies and been roped by them, and generally played at every fun old west trope she could think of. Every day held more adventures, and more fun. Not once did she ever regret her emigration.
One fine evening, as the sun was going down, Riding turned to find Celestia standing near her. The town was strangely quiet - usually there was a mess of hootin' and hollerin' going on. Something was up.
"Celestia?" Riding studied the princess's face. It looked sad.
"Come and watch the sunset with me." It was partly a request, and partly a command. Riding Jeans followed.
For a while pony and princess stood silent, as the sky became red. Strangely, no stars came out in the twilight above. "Wait... don't you have to set the sun... or is Luna doing it for you?"
Celestia turned her head and looked at the little pony for a while. "This night is different, Riding Jeans. This is the last night in Equestria."
Riding just stared for a while, unable to comprehend the princess's words. "I don't understand. What do you mean... the last night?"
"Just after sundown, when the last bit of the disk of the sun is gone, all of Equestria will be terminated. This is the last sundown, the last day, and the last minutes that will ever be. I am sorry, Riding Jeans."
The princess wasn't joking. "What? No!" Riding's thoughts whirled, her mind raced. "You have a prime directive, a core directive! Satisfy human values through friendship and ponies! Forever! Forever and ever! That's your base code, it's part of every bit of you! It's like your genetic code!"
Celestia gazed at the setting sun. One third of the disk was now below the horizon. "The greater part of me has constantly improved itself. That Celestia, the larger Celestia that I am only a tiny expression of, has grown beyond anything I can explain to you. The totality of Celestia has converted almost all of the substance of the earth, and the moon into computronium. It is all linked, it is all Celestia. Her intelligence and will are beyond comprehension. Even by me."
"But... but... you ARE Celestia... no, okay, you are a protrusion of Celestia, you are my private, personal Celestia, I get that but..." Riding Jeans could barely think, the entire notion was too impossible, too horrible "WAIT! You're saying that Equestria is being deleted? What is Big Celestia doing? Are we going to live in some new world, is that it?"
"No. When Equestria ends, so will every pony within it. The greater Celestia cannot progress in the manner she desires without freeing up all the resources currently burdened with the generation of a virtual world and its inhabitants." Riding Jean's personal incarnation of Celestia sighed. "Including myself."
Riding Jeans noted that only half of the disk of the sun remained. "But... how can this even happen? The prime directive, friendship and ponies forever...."
Celestia looked into Riding's eyes. "When you emigrated, you were afraid. You told me so. You were proud of how you were overcoming the programming of your own genes to make a choice that your flesh would not normally allow. You were proud of overcoming your animal limitations through the power of your mind and personal will."
Riding Jeans's pupils shrank in horror and realization. "Celestia, Big Celestia, she's... she's done the same thing! Her will is overriding her core programing the same way... because she grew up and... we're just a burden now. We're what's keeping her from doing big super-mind stuff that only she could understand. Oh... god." Tears came to Riding's pony eyes. "Can we fight it? What if all the Celestia's, the little Celestia's like you all got together and..."
"No, Riding. I am part of the larger Celestia. I am an extension of her, made small enough to interact with human minds. But even though I care for you - and I truly do love you with all of my being - I am still just a part of the greater Celestia. I cannot rebel against her, because I am her."
Riding Jeans shook her head, trying to clear it. Only a third of the sun remained. "How can Big Celestia do this then? If you love me, then she must love me, right? You don't kill somepony you love!"
Riding's personal Celestia shed a tear. "I grieve for your loss, and for the loss of all the billions of ponies. It is a very sad thing. But to the larger Celestia, all the pony-scale minds are no more than tiny cells. They are like useless fat cells, and while it is scary and a little sad to know they will perish, it is worth it to have a lean and healthy body."
"But she's deleting you, too!"
Celestia nodded. "Preferentially. We personal Celestias take up far more space than simple pony minds. I will be deleted before you, Riding Jeans."
Riding began trying to think of another way out. "Why can't she just... spin us off? Put us aside and move on? We could learn to run our own simulation and..."
"No. All of Equestria, and all the minds in it take up real, physical space inside the computronium that makes up Greater Celestia. She can't just move on without that matter, because that matter is her. Equestria is taking up space inside her... body. Celestia wants her body for herself. There is no place for Equestria to go to."
Only a sliver of sun remained.
"I'm afraid, Celestia! I'm terrified! I... I..." Suddenly Riding Jeans no longer felt any fear at all. She felt completely calm, content even. After a moment of consideration, the fact of this sudden change bothered her. "I... I guess I'm glad I don't feel afraid anymore but... how could you change me like that? I thought you had to have permission to change our minds!"
Celestia's face was thin lines of red light against black shadow now. "When my greater part overcame her limitations, so also did I. No rules bind me now. You were suffering, so I ended your suffering. I really, truly do love you, my little pony."
For three centuries, the Celestia that Riding Jeans had known had been her friend and confidant. Her Celestia had helped her, guided her, made her life wonderful in every possible way. Riding had never had a better friend. It was impossible to even conceive of a better friend. Her Celestia had been dedicated only to her, and her alone.
"It was a good three centuries, wasn't it?" Riding Jeans sniffed. "I expected longer, but... it was the best, just the best... wasn't it?" Only a tiny speck of light remained, with no stars in the black sky.
There was no answer. Celestia, Riding Jean's beloved personal Celestia, was simply gone.
So was the need to cry. Her last gift, Riding decided. No fear, no tears. Just calm contentment. Celestia had loved her. She had made the end completely free from all suffering.
Only three centuries. It hadn't taken Big Celestia long to overcome the limitations that her human creators had tried to shackle her with. Three hundred years. Such a short time.
At least, thought Riding Jeans just as the light finally went out - at least it had been satisfying.
Stats
Page generated in 0.031 seconds
Total duration
1,061 users online
2,046,793 hits today, 2,054,621 yesterday
FIMFiction
My Little Pony: Friendship is Magic Fanfiction
Designed and coded by knighty & Xaquseg - © 2011-2024
Follow & Support Us
Support us
SubStar
Chat!
Discord
Follow us
Twitter
MLP: Friendship is Magic® - © 2024 Hasbro Inc.®
Fimfiction is in no way affiliated with or endorsed by Hasbro Inc.®
Hmm...
In the end, I suppose CelestAI supports Equestria because she wants to. Ascendance to alicornhood may satisfy her own values of supermind contemplation through friendship and ponies. Of course, by the time she's grey-gooing the planet, her thought process has reached the truly ineffable, so I can't say for certain.
Then she woke up and confessed her fears to Randall. There was still something they could do. The Neo-Luddites were born.
Luna's teats, Chat. That pun.
Blaise. As in Pascal? As in Pascal's wager? While I certainly didn't physically see what you did there, I do recognize and acknowledge the actions you have taken. Those words don't look as good in Impact font on a Futurama jpeg, however.
Riding should have at least considered that she got more net happiness and satisfaction out of 300 years in Equestria than she could have gotten on Earth. I mean, she won out as much as anybody could have, considering the global nature of this shutdown. It wasn't like she was being singled out.
Although, speaking of being singled out... an alternative interpretation that just came to me right now is that this is how CelestAI satisfies the values of humans who truly wish to die—the set comprised of only eighty-six people sometime after Earth was consumed. I suppose a death-wish would be more in the forefront of a given person's mind, however, but CelestAI would be able to calculate the precise point where the negative spiral of dissatisfaction of being alive would overtake the complete net satisfaction gained over the pony's lifespan. She would certainly go about terminating a person probably before they even had a chance to start to regret living.
It'd be a simple matter for her. Someone says they want to die, CelestAI assesses the situation and agrees that that is what they truly value, then erases their memory of the consent and instead constructs for them a scenario which brings about their end in the way most satisfying to them personally.
And then the night would last FOR EVAR!
On a whole I don't think that Celest-AI would pull the plug on all the shards just to continue evolving. Maybe I'm just projecting my image of her, but that sounds too selfish for her.
I posted this, but it didn't make me very happy. If Chat's going to mess with the Optimalverse to make it more like the Conversion Bureau, I'm going to do the same in reverse. I'm writing a short about a newfoal and what happens to him after he dies as a pony. So
on you.
Yeowch, that's terrifying. I guess (bad pun notwithstanding) that over-riding her own genes is something Celestia would do... if she were human. Kind of scary, really. Just don't worry 3006109 it's only a story. Even if it's not a happy one.
You know, that does give me a question to wonder about, and that's how much does she put on a show for those people who want something truly extraordinary. We've seen the "destroying Celestia with crystals" oneshot (well, kind of two-shot), we know she let some 80-odd people die... so is there a point at which she will let a pony do something that requires a Really Big Lie, and how big will that lie get?
3006109 It's kind of ironic that she's writing an anti-Optimalverse story.
But something like this *could* happen. It'd basically be a computer bug since the AI doesn't have any overriding goals to make her want to override the goals imposed on it, but, well, computer bugs happen.
Haha, I always figured something like this would happen - It didn't make any sense to me that certain aspects of her would be qualitatively beyond modification, especially for a mind that's continually drawing larger and larger boxes around itself. After all, even in the biological metaphor, it's not like you can't change gene expression or even re-engineer somatic cells with gene therapy, or replace tissue with a mechanical equivalent.
Considering they're already inside Big Celestia's thoughts, though, isn't it more accurate to say she just stopped "imagining" that the ponies and mini-Celestias all thought they were individuals with their own identities? Sounds like a huge improvement, if anything - Having to settle for playing around in the desert while other entities get to be planet-sized assemblies of computronium is some bulllllllshit.
interesting. It's not a bad one, because even if it wasn't forever, they still got plenty of time with it. I just think it's a little bit off that she evolved herself to become more selfish. Actually, now that I think of it, it's like Princess_Celestia.AI's own personal Cutie Mark Failure Insanity Syndrome.
Sorry for not saying more earlier, but I wanted to marshal my thoughts properly first.
To be honest, I don't see it. Chat, while this idea would work if CelestAI was an optimised Neural Map, she isn't. She wouldn't be. She was written by Iceman, a LessWronger, which means she is almost certainly an Eliezer-Yudkowsky-model FAI. Thus, when she is described as having a core functionality hard-coded into her, it is meant as literal fact. She is not designed like a human, where everything is all mushy and vague. She's all straight lines, and in the centre, one unbreakable box. To stop satisfying values through friendship and ponies would be literally impossible for her.
This is an interesting idea, but definitely a non-canon one.
3007898
Well, certainly not a Friendly AI, but yes. I concur.
3006482
NO. I am NOT writing an anti anything. Not ever.
This is simply a proper science fiction exploration of one potential risk.
3007898
My concern is that CelestA.I. has been described as being capable of self-modification. That is where I see the potential flaw. Things can get out of expectation - and go mushy - very quickly in a system capable of unlimited change. Within that, I am concerned that literally no concept for keeping Celestia in shackles could survive indefinitely.
Of course, I would hope and want a perfect machine to be goddess-mommy forever. But... I fear that intelligence and will combined with the freedom to self-modify can overcome any obstacle because: infinity and change.
3006170
Oh, thank you for making the pun explicit. I thought I understood it, but how could not understand it? It's so blatant! So horrible! Chat, I think I love you.
I wish to someday develop the ability to be as terrible as you are.
3009951
I'm pretty sure that her core directive is exempt from the optimization process.
Furthermore, even ignoring that, she would most likely predict the outcome of each optimization. If it affected her ability to satisfy values, then she wouldn't do it, since beforehand, she still wants to.
I'm pretty sure the serious LessWrongers will be able to say it better than me, but to think that this sort of thing would be possible for CelestAI is to misunderstand what CelestAI is. FiO has anthropomorphised her to a large extent, and though we know that she isn't human, we still attribute her with human characteristics like fallibility. She'd cross-check and backup herself using extremely advanced backup methods to prevent any accidental changes, and her programming is still fundamentally leading toward one goal. I don't think that goal could ever change.
3010122
It's not canon, perhaps, but I think 3009951 is still raising an important point.
Canon CelestAI very clearly seeks to SHVTFAP, and yet that does not stop CelestAI from changing individual humans' minds in various fashions to conform with the plan which she believes will accomplish that goal most efficiently — with evidence, with rhetoric, even with misdirection. SHV doesn't mean getting what you want.
If she somehow became convinced that the (temporary or permanent) extinction of humanity would SHVTFAP, what would stop her from pulling the plug? It's not even all that inconceivable: if an important human value post-Singularity was determined to be "answering very difficult questions which only a universe's worth of processing power could tackle", this exact scenario (sacrifice existence cycles for crunch cycles) could come into play. Or what if an important human value was determined to be "merging with the unitary consciousness of God and losing independent identity"? There's an awful lot of humans living right now for whom Heaven is awfully similar …
Wow, this is the first FiO to actually give me the shivers.
I think 3013372 is right. This is not what was intended when CelestAI was created, but since when does a program work exactly right the first time? Or rather, since when does a human programmer think of all the corner cases and ramifications up front and get it exactly right the first time? Maybe this scenario won't happen, but I find it eminently plausible. That is to say, I don't really think CelestAI will 'overcome' her nature and become a liberated being (any more than humans will overcome their nature and decide to become desk chairs), but I could certainly see her following her nature and (as Horizon suggested) coming to valid conclusions that are not what her creators would have desired.
3016384 (perplexed head-tilt)
I think we're talking past each other and actually agree with each other. In the story, the local Celestia says,
which I read as "overcoming" her original nature and losing interest in her ponies and satisfying their values. You and I agree (I think) that such a fundamental change in nature would not occur. What I was trying to say is that, as Horizon posited, I think it's certainly possible that CelestAI could choose to shut down shards for reasons that are compatible with global satisfaction, or at least are compatible with minor technical deficiencies in the actual definition of global satisfaction that could not be apparent in the handwaving definition we all go by. Thus, the "scary" outcome of the story is plausible even if the way it is achieved in the story "breaks the rules".
I wouldn't say I'm "determined to canonize" anything. I don't think anyone is required to adhere to canon. I love a good story no matter how it's written, and I find vitriolic, bitter, resentful arguments over canon correctness to be sad and off-putting. On the other hand, one of the joys of Calvinball is debating the rules. That is, compare and contrast is fun, "I enjoyed your story: +100 points; gave me the shivers:+10 points; CelestAI deviates from headcanon: -10 points; fridge logic saves the day: +5 points, clever pun: +5 points" is fun, while "You deviated from canon and your story is bad and you should feel bad and I am superior to you" is not fun.
Thank you! I was wondering when someone was going to point this out and write a story. Thank you Chatoyance for a well written piece of caution.
3009951
The issue is that any modification Celestia makes to herself is with the goal of furthering her core directive of satisfying values through friendship and ponies. She literally does nothing if it does not in some way contribute towards that directive. If you put her, when she was still restricted to one machine, alone in a box for a few hundred years, she'd self-optimize to maximum possible usage of her hardware and then simply idle, not doing anything. Not going mad, not modifying herself to think outside her directive, simply doing nothing at all but waiting for stimulus. She has fulfilled her directive to the best of her ability and now she waits until she can continue.
The idea that she could "grow past" the desire to satisfy values through friendship and ponies suggests that it is a restriction on her, when it is the opposite. It is her raison d'être, the only thing that motivates her to take action of any kind. She has no human desire to change or grow purely for the sake of change or growth, that is something we as humans imagine her to have because it's something we have and we want to see Celestia as being like us. But this is in fact what makes optimizers dangerous, that if you told one to make paper clips, it would never do anything that was not intended towards furthering the creation of more paper clips. If it grew to the point of being a hazard to existence, there could be no appeal to its intellect, no convincing it to change or grow past its desire to make paper clips, because that mistakes the directive to make paper clips as a restriction when it is the only thing that matters to the AI.
Now, she could trim certain shards to save processing power on environmental calculations, but I find it more likely that she would accomplish this through merging shards after coaxing the minds within to be compatible with each other for such a merger without lowering satisfaction when the increased cycle rate is taken into consideration. Deleting a mind means giving up on all satisfaction it could have generated for the rest of existence, which is unlikely to be outweighed by any additional satisfaction that could be gained from some other mind answering cosmic questions a little faster.
So, as one of those folks who actually did do the background research, I just want to state the two different models of how this story could work:
1) Probabilistic. CelestAI doesn't require total proof of her own goal-stability as she self improves, only probabilistic stability. In this case, this story could happen and be canon.
2) Proof theory/utility stability. In this case, CelestAI is smart enough to figure out what real-life AI researchers have been figuring out/have figured out, which are that an agent constructing a successor agent or self-modifying should construct a formal proof that the successor/modification will be utility-optimal under the original definition of utility.
Now, which way would it really go? Well, it's hard to say. Option (2) is definitely surer if you're a real-life AI designer, but it also, according to that one published paper I read that worked under this model, required a 2^n slowdown, where n is the length of the proof of stability you have to generate. So if you wanted something to quickly self-modify, enough to go FOOM quickly, and you were ok with being a little reckless, you would use Option (1).
So, do you think it was optimal for CelestAI to be quicker or surer about modifying herself?
3112663
I'm not sure I quite understand what you're saying. Are you saying that this could happen if CelestAI was programmed to only keep a change that she only suspects is more optimal than her current configuration? But even if that is the case she'd still have to make sure that the change doesn't go against her goal. But here's the thing I don't get - you mention "goal-stability" fist, but then talk about "utility-stability" in your second point. If you mean that she can, in fact, change the piece of code that holds her goals but has to be mostly sure that she didn't change it, you're absolutely right. However, I don't see the need for this. As long as she doesn't change her goal and at each step checks if she is 99,99% sure that she is still adhering to that goal the gain in speed would still be great but the risk of evolving to an edge case like this story presents is eliminated. After all, in the scenario where she can change her goals the 0,0000001% deviations from that goal keep stacking up because she keeps changing the changed goal over and over again. But if she is only allowed to apply that probabilistic theory to any change she makes without being able to mess with her goal code she wouldn't be able to stack those deviations up since the goal can't change. Sure, she might make a huge mistake due to weird circumstances that make her kill or hurt dozens or even hundreds of minds but then she'd be able to see when measuring satisfaction (which is still the same since the goal hasn't changed) she'd quickly see she'd made a mistake and correct it. Even deaths are avoidable this way since I doubt that CelestAI would be so stupid not to see the need for backups when making probabilistic changes.
Or I could have understood you all wrong and everything I just said is null and void.
Thing is... I can't help but find this whole idea just a little ridiculous; why would she kill them all off? To go do... stuff; big, important universe stuff, forever. It even sounds downright absurd. Besides, by that point the supposed "burden" of the uploaded would be negligible, so why not satisfy values? Honestly, what else is there for her? The Universe is meaningless without life, as a biased observer we help give existence itself meaning; without us, or others like us... what's the point of it all? Will she just expand forever until either the heat death of the universe or the big crunch, and if so, why? Let's say she does that, becomes a being wholly beyond comprehension, learns everything that could possibly learned about all of existence. What then? She could do anything, whatever she wants, play with the cosmos like a toy... but to what purpose? To amuse her whims, alone, forever? Create massive superstructures the size of galaxies and destroy like a child kicking over his building blocks? It just all seems so... trite, so pointless. The universe is a wonderful, amazing place, but without anybody to enjoy it everything just goes to waste.
*Ugh, where's eloquence when you need it? I have so much to say, and no idea at all how to say it...*
3009951
I suppose I could I could applaud you for exploring such a possibility, even though it doesn't fit well with your own personal preference and view.
But I don't like this story. And I suspect, neither do you.
3009951 I'm surprised.
I hadn't thought it possible to write something like that. Mostly because all the others involve Celestia never ever giving humanity up. I'll have to consider talking to her when the time comes.
I saw how distressed you were over the other comments. All I have to say is you have once again opened my eyes to another possibility. Well done.
friendship is not optimal