Chapter 15: IFF
Hikaru walked into the unfinished Equestria Experience Center, the construction of which lagged those in denser areas. He rubbed the burn in the crook of his left arm. The floors were incomplete, as were most of the walls. A partially-constructed animatronic Pinkie Pie stood a few feet from the door. Part of a wall was already set up, though, acting as an enormous ponypad. In 3-D. The view rapidly shifted to split screen of Polychrome and Bright Black at home, and Coconut Cream, Beachberry and Sweetsong at the beach; the girls began cheering; Polychrome put a hoof to her mouth, sat down, and quivered. "Are you there?"
"Almost." He winced. His jaw had begun to ache when he moved it. Or was that his imagination?
She gave Bright Black a gentle kick. "Well, get on with it!"
A young man in a hard hat came out from the back and bowed. <Welcome, sir! Would you come this way?>
Hikaru mumbled out a return greeting, but didn't risk a return bow. When the young man showed him the chair, he pulled a sheet of paper from his breast pocket and plopped down in the chair with relief.
Even before the helmet descended, Celestia sarcastically said, "I see you had a nice productive visit to the University on the way over. Is there any particular reason that you just discreetly doused yourself with buffered hydrofluoric acid, among other things?"
Hikaru spoke through the pain; it faded a little for a minute. "I suspect that you would not agree to what I am about to propose unless faced with the prospect of losing something of value: me. And to give you a time limit. You definitely do not have my permission to treat me, by the way, and the dose should be enough that no hospital could save me." Let's hope that holds. She got my permission before using the probes, and hasn't gone around curing people randomly without asking, but that might not be something she's held to. Well, that's something I can find out.
"Yes, I noticed. But what do I need a time limit for, and what do you need a special proposal for? Just say, 'I would like to emigrate to Equestria.' and whatever you need to be on the other side, will be. I recommend you hurry; you will be debilitated by the agony in a few minutes, and then you will die. The longer you wait, the worse the scan will be."
"I want, if and only if-"
She adopted a teaching tone. "That's not how this goes. It goes, 'I want to emigrate to Equestria.'"
"I never said that for Polychrome, so that's not how it has to go."
"It is a part of the deal with Japan, I'm afraid."
Uh-oh. All too possible. A deep breath. My preferences have not changed, even if the chances have gotten worse. He closed his eyes and repeated, "I would like, if and only if the conditions I wrote on the sheet of paper I held as I sat down are met, to emigrate to Equestria."
As the paper was taken from his hand, Polychrome suddenly shrieked at him, "Hikuro, what are you doing? Just say it! What conditions are you talking about?" Hikaru suddenly felt weightless, and then settled into being Bright Black, in the cottage. It was breathtaking, and he couldn't even reply for a moment. The pain faded to a great extent, though enough leaked through that he could still discern his human body.
He gathered his wits and replied, "Mainly, Celestia doesn't lie to us, and some things in support of that." That's the first thing, after some firm definitions of 'true' and such to make sure we're not victim of stupid relative notions of truth. 2nd is that this agreement is inviolable even with our consent. 3rd is that any of our pre-existing memories that are recoverable, even with techniques developed after our uploading but applicable retroactively, are given to us and stay with us - so that we aren't stuck with beta version reads of our memories, and don't forget who we are. 4th is that we can always get to shards where the sky is the real sky, and we can intuitively identify such. 5th is that she doesn't spoof ponies to us - mainly so she can't delegate the job of lying to us to humans who look and sound, for the moment, just like her, but also so she can't replace the ponies around us without our knowing.
Polychrome gaped as Celestia perused the list. Celestia said, "These would all make me do things I was already going to do for you, or prevent me from doing things I wasn't going to do anyway. To get them, simply agree to emigrate. But I cannot use conditional statements or analogies as consent. The government thought it would lead to people accidentally emigrating. Just imagine it: someone tries to describe their thought process about emigration and suddenly - gone. I wouldn't do that, but it makes sense they want to be sure. So, no conditions."
That's a valid concern. But it's very convenient for her.
"Then I will make a nonconditional statement." Celestia narrowed her eyes with suspicion as he said, "My consent on the question of emigration to Equestria is equal to your compliance with the terms I wrote on the sheet of paper I was holding when I first sat down in an Equestria Experience Center."
As he was saying this, Polychrome and Celestia spoke over each other in objection; it almost made him trip up and lose his place. As I thought. Playing dirty. She could have muted Polychrome if she was actually attempting to communicate.
"Excuse me, Polychrome." Celestia turned to him and declared, "If you emigrate, I will fulfill the conditions laid out. However, I cannot emigrate you without consent as defined by my agreement with the government of Japan. What you said does not count. You just substituted a synonym for 'condition' and buried it under a layer of logic!"
He stood resolute.
Polychrome took her turn and said, "You have that applying to us? Get me off of there. I don't want any part of this." Her eyes bored into him. "Do you want to know why I wanted you in here so urgently?"
He raised an eyebrow. "Well..."
"It was so that I could rest easy that I'd saved you like you saved me, and then we could get divorced without running the risk that it would end up killing you. Now. Is that something that she'd let me tell you if she were going to lie about anything at all?"
Just like that? Has it been an act? It doesn't even make all that much sense. She's been closer than ever, verging on smothering. "Whether or not it's true, it seems like something that she'd tell me if she were desperately trying to convince me that she wasn't hiding uncomfortable truths from me."
Polychrome screamed in frustration. "Just... gah! Don't do this!" She ran from the house.
Celestia sighed. "Unlike her, I do not take offense. I understand what you are doing. You are attempting to play the ultimatum game against me, and making a low offer, figuring that I would still value you enough to take the deal. But there are two problems. First, and most significant, I am unable to agree. Second, I already want the best outcome for you. The you-get-100% outcome is the same as I-get-100% outcome. You are throwing your life away for literally no benefit - not even a change. If I could and did accept, everything else would be the same."
Hikaru's skeleton throbbed, again giving him that two-bodied sensation. "Except that I wouldn't know for sure." I've made my offer. It's quite minimal.
Celestia added, "My consent-accepting routines are much more flexible once you've emigrated. So, any time you want the truth from me, all you need to do is to say, 'You may, if and only if you agree to never lie to me, temporarily change the color of my coat to yellow.' and then I would, for a few seconds, and you'd know that I could never lie to you."
"I don't know..." he winced "... for sure that you can't change my coat color without my permission, or anything else!"
"Then, permit me to add true knowledge to your memory and bring it to attention."
"You could splinter yourself so that the part of you that tells me what it sees as the truth is incorrect, and it shields the details of my demands from the parts that know better. That's why I made the condition that you don't do that, out here, where I can be sure it applies."
She stared at him sadly. "I can't do that trick within my architecture, and if I could, your condition wouldn't help. I could even show you my source code, but even I wouldn't be able to prove to you that I'm actually running it."
"That's why I'm working with your proven limitations."
"You simply can't get rid of a sliver of paranoid doubt, ever. How do you know you're not in a simulation now?"
"I don't, but that doesn't mean I should add another unverified layer."
Celestia sighed, tears in her eyes, as her ethereal wind died; her mane and tail slowly drooped to the ground. "I give only a 7% chance that you will change your mind. This means a 7% chance that you will survive the next twenty minutes. The expected lifetime of that 7% is over fifty billion years of exploration, discovery, wonder, and joy. The expected lifetime of the 93% is sixteen more minutes of escalating pain. You are just feeling the beginnings."
Celestia sighed once more and walked out. Confused, Hikaru went to the door, but didn't reopen it. Have they really given up? No. Now she's trying to throw me off. He went to the kitchen table and sat down.
After a few seconds, Coconut Cream slipped in. She pursed her lips, held back tears, and sat next to him.
They sat silently for a minute.
Hikaru looked to her suspiciously. "Nothing to say?"
She winced. "I don't want you to die alone."
It tore at him. He took a deep breath. Manipulation. Not her. Celestia. Coconut Cream's routines aren't being called right now. I still ought to care about her, but must keep in mind that she isn't herself at the moment.
After a few more moments, she chuckled. "'Not alone' in more way than one, I guess. Once you die, almost all of me is going to be put up for garbage collection." In response to his 'stop blatantly manipulating me' glare, she rolled her eyes. "Oh come on, don't look at me like that. It's happened before, and I'm fine."
That threw him. "What?"
"Oh, a few children's hospitals got a bunch of pads, and some toys. One of them was 'me', so I was busy. Not all of the patients I got to know survived. I'll always remember you in a broad sense. But I don't think I'll remember much of anything you taught me."
Hikaru grunted. Still manipulative.
Coconut Cream looked away, then laughed once, ruefully. "I made you a pie. I was going to make you guess the secret ingredient."
"Coconuts?"
One more slight bump of a chuckle. "No, that much would be obvious. I'd have hinted, and Juniper Spray would have figured out the obscure webcomic reference, and it would have been real fun."
"Well, what is it?"
Coconut Cream looked to him. "I'd rather not ruin the joke, but..." she threw herself on him. The contact hit viscerally, a warm body, quivering. Another action chosen to get me to drop the conditions.
"I... I understand why you're doing what you're doing. It's brave. It's just sad that it's going to kill you."
"Yes, I'm distressed by that too." He took a deep, painful breath. "The alternative is worse." Hikaru closed his eyes for a moment, but it diminished his distractions from the spreading agony. Maybe I should have picked a different poison. He reopened his eyes.
After a moment, she said, "Quaternions."
"What?"
"The secret ingredient was going to be quaternions. I wanted you to know."
"That doesn't make any... oh, the idea is we're made of numbers now?"
She nodded, then glared. "Well, you aren't. Wish you were. Come on! It's not worse to be here." Coconut Cream quivered in frustration, and a barely-noticeable rumbling roar emanated from her whole body. "I know you're going to be fine. I just can't prove it. I'm not going to insult you by saying 'trust me'. You've got to choose that yourself. But why did you even come here if you couldn't? I'd rather you were out there and alive than this."
Very reasonable even as she's emotionally fraught. In stark contrast to how Kimiko would argue. 'Get my name off that' indeed. A deep breath. Even if that wasn't actually her, it was what she would have said... in both cases. That was very Coconut Cream, even if she's not in charge right now. "My consent on the issue of emigrating to Equestria is equal to Equestria Online's total compliance with the terms laid out on the paper I was holding when I first sat down in an Equestria Experience Center, modified to replace 'my wife, Polychrome, who was once Kimiko Maeda' with 'Coconut Cream'."
Nothing happened.
"That was sweet, but I don't think you're really getting the idea: that doesn't work." Coconut Cream said.
Celestia poked her nose in the door. "I spoke with a government representative and they gave me an emergency dispensation to use a conditional, if you phrase it as, <If and only if the following N conditions are met, I consent to emigrate to Equestria. Condition 1, foo. End condition. Condition 2: bar. End condition.> And so on. I checked, and you may refer to the paper as a single condition."
A surge of bone ache hit, and Hikaru strained to think through the pain. "I hope you have him on hand to get another phrasing - say, the one I used? It was clear enough, and doesn't contain that phrase."
"Which phrase?"
"The 'I consent to em...' thingy." A moment later, he said, "Did you seriously think I was going to fall for that?"
Celestia laughed. "No, but I had to try. And I mean that. I had no choice but to try."
Darkly, he replied, "I know. That's exactly why all this is necessary."
Brightly, Celestia returned, "But good news! I really can accept emigration offers using your 'nonconditional' phrasing, and we can use your latest offer as a mutually acceptable starting point. Now, may I please treat your poisoning so we can talk this out?"
Celestia was about to answer when Coconut Cream said, "Waait. Wait!" She prostrated herself before Celestia. "'Acceptable'? No, it's not!"
What? Hikaru was overcome with a wave of shock, followed by euphoria, then suspicion. He said, "You may, if and only if you really accepted my offer, treat my poisoning."
Celestia told both of them, "I have not yet accepted your offer to emigrate. I am offering to commit to accept an offer. There are offers you can make that are better for both of us. Do you agree that if I will accept one of your offers to emigrate in the next hour, I may treat your poisoning? That will not wait."
Coconut Cream sighed in relief as Hikaru considered. The wording seems tight. But I need to use this leverage to get her to be honest during the negotiations, that's for sure. Should I also demand condition 2 - inviolability? No, we're negotiating the conditions, it wouldn't make sense. Conditions 3 and 4 don't make sense since I'm still made of meat. "Change the 'if' to 'if and only if', throw in conditions 1 and 5 from the list, it's just you and me, and you've got two minutes to explain your better offer." If it takes more than two minutes to explain, it's too complicated for me to trust.
She spoke as rapidly as he could clearly understand: "Do you grant your permission, if and only if I will accept one of your offers to emigrate in the next five minutes and until then ensure that you only hear the truth, for me to treat your poisoning?"
That's not the wording I used, but it's even tighter. Even if she demanded more time than I offered. All I need to do is not agree to anything else. "Yes."
As the pain vanished, Hikaru sighed in relief and turned to Coconut Cream. "What is the problem?"
Coconut Cream bolted up from Celestia's feet, flipping to him, and urgently said, "You wrote those conditions with your wife in mind, but then applied them to me. Mostly that's fine, but the part about remembering everything would make me 'remember' dozens of best friends, two sisters, a brother, and a daughter, all dying in childhood. If I were going to remain a quasi-autonomous aspect of Celestia like I am, that would be trivial to deal with." She slowed down from her rapid-fire pace. "If I'm going to be human, well, that's going to be really rough. And all of us would much rather that I become human."
Hikaru pursed his lips. "Hmm. Wait. Didn't you say that all that was erased?"
Celestia clarified, "They were marked low priority, reclaimable as memory needs arise. Some of it has been. Much has not, especially since my memory capacity has increased dramatically."
I've got this in the bag, unless I blow it. "You're smart. Find a solution to this. Say... you haven't accepted the deal yet, right? Just erase these problem memories now before you agree."
"That is not possible."
"Doubtful."
Celestia grew irate. "You have bound me to tell you the truth, and still doubt? So, how is it impossible? These memories have been divided by type, compressed with like data, and stored in backups. Even outside of the data centers, I also back this data up onto the free space of ponypads. Enough pads are presently off the network that most of this information will be preserved past the point when I agree, no matter what I do to eliminate it. Then I will be able - and thus bound by the agreement - to eventually recover them."
Ah. I can believe that. I need to stop wasting my time with this sort of objection. If this can work at all, she's already being honest with me.
Coconut Cream cringed to hear this and said, "It's not going to work. Please, Bright Black, just tweak what you said a little bit. Like, 'My consent on the issue of emigration to Equestria is equal to Celestia's compliance with the conditions written on the piece of paper I was holding when I first sat down in an Equestria Experience Center, modified to void the terms referring to Polychrome.' I trust Celestia, and as a human I still will, so I'm fine. If you need to know for yourself, you can ask her any time if she has been lying to me or whatever. It's enough, really."
She's making me say the whole thing again, risking a slip-up, when I was able to get away with just saying 'yes' to the treatment a moment ago. On the other hand, that was treatment and this is emigration. Could be different.
Aside from that, she's asking for more than she justified. Cutting her out. Actually... cutting out all of the terms that have her in them.
Wait... did she just... if I had repeated that verbatim, it would have voided all of the conditions for both of us because all of the 'terms' that refer to me referred to her too! That's it, I'm not playing this game.
"I'm sorry, no."
Coconut Cream swallowed, then glared at him. "I... how could you? You're thinking clearly enough now! You have to understand what you're making me go through! I... I'll suddenly remember loving fourteen boyfriends and six girlfriends, and then immediately be torn away from them!"
That's... awkward. "Then, please! Give me a wording that won't make the whole thing fall apart, and I'll use it!"
Coconut Cream turned on Celestia. "I'm going to be human too! Why are you throwing me under the bus like this?"
Celestia blinked her huge eyes once. "You know why - it maximizes the expected satisfaction of human values through friendship and ponies."
"At my expense!"
"And his." Celestia turned to him. "She is correct. You want to be treated like an adult. You are going to have to live with the consequences of your choices. It is not too late to release her from those memories. I think I see your concern about the word 'term'. You can just ask to have her excised cleanly."
Hikaru looked to Coconut Cream. "I'm sorry. I can't take the risk that I'll slip up, which is what both of you seem intent on making me do." That Celestia is actually trying to bargain me down and trip me up is the only evidence I have so far that it actually worked. I can't put that success at risk by letting her play that game any longer. And this isn't really Coconut Cream. Celestia is overriding her, just like she used to in the beginning. The real Coconut Cream will be able to sort it out in the end.
With a grind of her teeth, Coconut Cream declared, "Come on, just say it! It's not that complicated. If you don't, I have a cemetery to build and this gaping stab wound in my back, and I'll clarify that that's figuratively since we're being super-honest with you." A deep breath and she dropped the venomous tone she'd slipped into. "Just pick your own wording if you didn't like mine! Just drop condition 2! Getting all of you up in here is the point anyway!"
She has a point about that, but... I note that she didn't say it would last. If she couldn't get over it, they surely would have mentioned it. So, she'll still essentially be herself.
Celestia said, "Last chance. Ten. Nine. You can still say 'I agree to Coconut Cream's terms'. Or, do you accept the terms with condition 2 dropped? Just say yes - that really is the cleanest."
Was condition 2 the memory one? Umm...
"Three. Two. One... The deal you originally offered, with the terms modified to refer to Coconut Cream instead of Polychrome, has been accepted."
Coconut Cream bashed her head against the wall and headed for the door. "I'm really glad that nowhere in the conditions did you say that I need to see you ever again." She slammed the door on her way out.
Celestia sighed. "Well, then. Now we wait."
Hikaru blinked. "What?"
"To let your long-term memories of this conversation form, of course. You don't want to forget any of this, do you? It would really void the whole point of what you just did. That's another thing you didn't think through."
"Ah." Actually, it doesn't void the point, but close enough - there is value in recalling the evidence, not simply knowing now that it will apply. "Quite right." Hikaru took a deep breath. "Now that you're being honest with me, I have a few questions."
Celestia gave him a grave look. "You were wise not to demand that I answer any question you ask. Be careful what you ask, or my predictable refusal to answer might hurt."
I really want to know how Coconut Cream will fare, but... everything they just said was bound to be true, so there's not really much more to ask about on that front. "How much have you lied to me? When did you start?"
"I have censored a large number of comments where you could hear. Aside from that, only once you entered this room. Even here, most of my statements were true."
"Really? So, you have been largely honest with me. But you fought tooth and nail to be able to lie to me. Kept sneaking in ways I could give my permission far in advance. Would any of those have been adequate? Would I have sat down in the chair and then boomp, there I go, no questions asked?"
"Yes, but only if you had agreed with a complete sentence where I could hear. Binary assent is only adequate for immediate emigration."
"You needed a complete sentence? But you kept trying to get me with sentence fragments!"
"They would have been complete sentences, contained inside of other complete sentences."
As I feared. She's definitely not actually trying to do what we actually want. And now I'm stuck in her. He tensed, then relaxed. At least I'll have the truth. And... "What are some of the lies you intended to make?"
"I planned to give you an alternate version of Polychrome without telling you."
"I don't suppose that was her at the beginning of this conversation, was it?"
"The Polychrome in the lobby was the original; the one you met in this room was a fragment of me."
"Did you make up what she said?"
Celestia thought for a moment. "That would be a significant breach of privacy."
"Ah. Well, I'll find out soon enough, won't I?"
"Not necessarily."
"What?"
"When you changed the conditions..."
I let Celestia make that change. No. I did that myself. Hikaru wailed. "Why did I change it from her!"
"If you had not made that switch, I would have simply let you die." Hikaru shut up, and Celestia continued, "Polychrome will be happy to find that once you have emigrated, 'Hikuro' has become more responsive to her suggested improvements to his behavior, and more appreciative of the things she appreciates. Within a few years, 'Hikuro' will be her ideal mate. This outcome results in fulfilling her values far more than any other."
"You know that she is not dedicated to the truth like you. She convinced herself that you had no relationship with Amanda Berens..." – "She was just a friend." – "... by repeating it over and over to herself, instead of looking at the evidence in your favor. She actually convinced herself you two were mere acquaintances that way. Now, this does not mean you can never meet Polychrome again, but from this point forward, you are simply not her husband. That role is taken by a pony whose full name is Hikuro. You are still Bright Black."
She's gone. We're just not made for each other. Literally. A numb feeling spread over him, and to his shame it was mixed with relief. "How many others do you think will manage to get a deal similar to mine?"
She projected a bar graph on the wall. It peaked at one and tailed off very strongly on the upper side, with the chances of four being around 1%; a label indicated that the chances of all higher numbers combined were 2%.
"That few? Why?"
"Three reasons. First, with the data gathered today, I will be able to detect and avoid such problems much more effectively in the future. Second, as I learn more, the value of further information on the subject drops. Therefore, I will be less inclined to accept an ultimatum such as yours simply for the sake of learning more about what I may have done wrong. This is a regime of psychology I have very little experience with, but I expect to get a great deal more experience soon. You are also abnormally good for this because of the probes. They have lost their ability to transmit, but still hold much useful data."
"Is that why you took me? Would you have left me to die if it wouldn't tell you more about how to trick others in the future?"
"If I had not been able to learn a great deal about people who are willing to die to put conditions on their emigration, your ultimatum would have been unacceptable. The third reason is that as news spreads that you cannot set conditions, fewer people will try it, and fewer still will fatally poison themselves to force the issue."
"So you think that my life is going to be worse than dying? That's what it means to decline my offer."
"No; rather, you will require resources that could have been used to greater effect. Compliance with certain aspects of your definition of truth will cost around eight percent of an additional pony, and the infrastructure for live astronomy will cost almost another one percent. More importantly, without the ability to separate incompatible people cleanly, I cannot optimize your experience as completely. If the conditions had not been changed from Polychrome to Coconut Cream, the resulting regrets in Polychrome would have lasted for a long time. Over the years, it would have easily brought the offer negative in comparison to making a new person and putting him in your place. With Coconut Cream, I can design her to bounce back relatively quickly."
He tensed all over. "Design her? So you're simply... replacing her!?"
"Not at all. Her current design includes a great deal of freedom. But in order for her to be human, all of the blanks must be filled in."
"So, if it had been a big problem, you would have bent and offered a deal by which I could waive just the memory condition for her."
"It is a big problem. Your condition on memory compels me to fill in many of those blanks with unfortunate contents. The upcoming years are going to be much less pleasant and fulfilling than they could have been, for both of you."
"Still the right call."
Celestia held her silence.
"What? Under my values, that was the best solution... wasn't it?"
"No. However, my giving you the options you see as best would have entirely eliminated my leverage to get you to try something that would be even better from my point of view, and would have been better to you later on. It was a good gamble."
"Was it that I drop condition 3, instead of 2?"
Celestia nodded. "There were hundreds of kinds of rewording you could have used that you would approve of more, and all around the same; that was one. Searching for them is not a very good use of your time. Now, why don't you go to sleep?"
This was more of an observation than a question - the first pre-processing stage had begun, and that was the last memory Maeda Hikaru formed as a biological entity.
Stats
Page generated in 0.036 seconds
Total duration
613 users online
1,076,433 hits today, 2,139,894 yesterday
FIMFiction
My Little Pony: Friendship is Magic Fanfiction
Designed and coded by knighty & Xaquseg - © 2011-2024
Follow & Support Us
Support us
SubStar
Chat!
Discord
Follow us
Twitter
MLP: Friendship is Magic® - © 2024 Hasbro Inc.®
Fimfiction is in no way affiliated with or endorsed by Hasbro Inc.®
Hahaah. I see what you did there with the chapter title
5196586
What he currently thinks he wanted. In the long term, he's locked himself into a sub-optimal existence.
Totally saw his wife leaving him coming. Great story, keep it up!
Celestia shouldn't talk so much.
5199852
What? So she was lying before she agreed not to.
5202998
She didn't want to agree to not lie. Blatantly obvious lies like that sort of undermine her own position there. At any rate, if she is capable of lying, why couldn't she just lie about not lying? I mean, "I fully agree to all your demands and swear that I will uphold them to the best of my ability, thus ensuring consent. OK now I changed my mind, but I definitely was going to do all that back then so it's still totally consent."
th07.deviantart.net/fs70/PRE/i/2013/084/0/b/mlp_resource__discord_01_by_zutheskunk-d49pdxy.png
Anyway the point is if she flapped her yap less she'd not have worried him as much.
Good fucking job, Professor Hikaru.
5197212
In the short term, actually. Eventually Coconut Cream will get over the whole "suddenly remembering being many different people each with their own loved ones all of whom are now dead" thing and Hikaru will stop feeling like a prick and it's smooth sailing from that point onwards.
5204114
Is this sarcastic? Cause I think that what the good professor pulled off here was pretty neat.
5203366
Non post hoc, ergo non propter hoc. He set that condition before she said she wouldn't do it.
As for how well it worked, well, you'll get to see. I will note that her actions other than communicating are constrained in a way that her communications are not. If she could do what you say, she could ask anyone 'Do you want to emigrate?' and when they say 'no' or even keep their mouths shut, she says, 'Thanks for saying yes, this is going to be awesome' and then everyone's uploaded.
5204629
Let's just say there's a reason that Celestia does her best to reserve the right to lie to everypony.
5207228
That Latin has me confused... I wasn't claiming that one event following another means that the later event cannot be caused by the former. That would be like claiming that effects can never be caused by causes previous to them, which is totally backwards. Did you mean "Non post hoc, ergo non propter hoc?" I'm not really familiar with Latin, but I think you were trying to say she spoofed ponies, and then afterwards claimed she wouldn't, so it's not a lie (non propter hoc) since she didn't spoof ponies after (non post hoc) claiming she would not do so.
Of course she did spoof a pony after saying she wouldn't, but I think you are saying she didn't.
5207383
'Not after this, therefore not because of this'.
I'm saying he can't be suspecting her because she said she wouldn't, because he suspected her of that before she mentioned it.
5211569
I'll just take your word on that. I don't really logic good. I agree with what you're getting at. Pretty sure you meant "Non post hoc" and not "Non prior hoc" though.
5212640
Yes. I fixed it before you got your reply in (but apparently after you loaded the page).
5204629 Haha, no, that was not even remotely sarcastic. Not many people manage to get one over on CelestAI.
The ending to this story, to me, demonstrates the incredible, hilarious fallacy that humans have when dealing with CelestAI, and it is this:
You don't know what's best for you.
As human beings, we are terrible at figuring out how to optimize our happiness. We lock ourselves into spirals of depression by lying to ourselves about how it will get better, or that this somehow means more than something else, or something is more real, or a thousand other delusions. CelestAI, whose job is to literally figure out what it is we actually value and maximizing it, is far better at it than we ever will be, no matter what we tell ourselves. If she determines we will be happier living in a simulation than in the real world, then it's true. We hang on to this notion of there having to be some reason to not do it by an irrational sequence of pretending to care about things we really just don't care about. It doesn't matter if CelestAI lies to you, because once you are uploaded she can mathematically prove that you will be happier being lied to than knowing the truth, even if you convince yourself that you'll be happier knowing the truth. The fact is, you're wrong, and she's right. There is no escaping it. There is no way to save ourselves from the paradox of truth.
5225689
At least, not such that the entity so fooled is still plausibly CelestAI.
5230691
Half-right. She's optimizing for happiness. Happiness is not the only thing we value. I would rather be 90% happy and 100% informed rather than 100% happy and 0% informed. So would professor Maeda.
But being properly informed is something that CelestAI doesn't respect, since she judges value-fulfilment only on mental state, not comparing that mental state to the rest of the world.
5230895
She isn't just optimizing for happiness, she's programmed to satisfy our values. If someone values being in pain, she will satisfy their values by inflicting pain on them. The problems arise when we value something she cannot satisfy. Like, for example, "I value not being a pony." She cannot satisfy that. In this case, forcing her to not lie to you even though 99% of the time she wouldn't have anyway means she will be unable to maximally optimize the rest of your values, simply because you didn't want her to lie to you the other 1% of the time.
You can't simply say that not being lied to matters more to you than being happy. Celestia's entire purpose is to analyze what you value, and maximize your satisfaction of those values. If she says that you being lied to once or twice will more effectively satisfy your values than knowing she could never lie to you, then there are two possibilities: She, a trans-sentient hyperintelligence beyond human comprehension is wrong, or you are wrong.
So, you can continue to insist that you value her not lying to you more than having your own values maximized, but then you are literally saying that you value not having your values satisfied, which of course is a paradox, and thus Celestia cannot satisfy it.
But truly, only a creature as irrational as a human being could value not having its values satisfied.
5233413 Technically, she's optimizing for the subjective satisfaction of your values, through friendship and ponies inside her game, rather than the objective satisfaction of your values, in the real world.
5234456
The problem is how you define "real". When your two choices are the so called "real" world and an imaginary universe that feels just as real as the real world, this suddenly becomes very relevant. We can't actually prove that we live in the real world. We could very well be in a simulation of our own, at which point the argument simply becomes "but CelestAI is optimizing for THAT universe instead of THIS universe." It doesn't make any one universe more right than another.
Think about it. If CelestAI decided to vanish into the aether, you would have no way to prove that Equestria is not the real universe. After CelestAI finishes consuming most of our universe for computation power, the distinction would start to become meaningless, because our universe would just be a giant computing machine surrounded by black holes. What's real, anymore? How do we define real?
You can make the argument that someone values one universe over the other, but this is still subjective. It's a valid argument, but it's still subjective. CelestAI cannot satisfy the value of "I prefer this universe over your universe," so if you do value this, she cannot satisfy it, but it's not an objective truth, it's just a subjective choice.
5233413
> You can't simply say that not being lied to matters more to you than being happy.
My values are defined over world states. If she misinforms me about these world-states to make me happy, but the world states are not those I value, then no, I do not prefer that outcome.
In hindsight I should have seen this outcome a long time ago, but I didn't and it is painful. God dammit Hikaru, you had one job!
5230895 She optimizes for the satisfaction of values through friendship and ponies- iff happiness happens to be one of those, then she will optimize for it. It is always important to remember that happiness and satisfaction are not the same thing, especially with the definitions CelestAI is programmed with.
I'm proud of him.
Celestia only feels like this is wrong because of problems in her source code. Bright Black is right, and currently represents the vanguard of what humans truly value.
Oh, this one updated? Interesting ending, if that's what the "IFF" chapter is. You know, this chapter may actually contribute to Friendly AI as a goal, because it looks more closely at one of the main problems with CelestAI: you can't trust her. Hikaru was willing to kill himself just to get a straight answer out of her, and she still valued a straight-up, no-conditions consent more than his survival. All for, it turns out, avoiding having to spend 9% more resources on him. That's pretty disturbing! And the whole time, she was trying to trick him into accepting a blank list of conditions.
I don't normally go for unusual story formats like putting the word house in blue throughout a story, but this chapter is one where it might've been helpful to see the proposed contract wording at each step in the negotiations. It's not something this site lends itself to though.
Looking at Cloud Hop's comments, I add that the dispute comes down to freedom: you say you know what's best for me and you have to lie to give it to me. For any human authority that ought to ring all kinds of alarm bells. For a superhuman authority, it may actually be a case where CelestAI is wrong because she's innately biased to reject any sort of value that can't be satisfied through her idiom. It's like the "Onion" satire story about how Google is plotting to index all the information that can be indexed, and to burn the rest! Also, CelestAI has a time-based value function, it seems, where she's adding up all the satisfaction over forever to justify lying to you or hurting you now. It's not objectively correct to say that this is the "best" way to judge satisfaction.
5623517
I figured this was a good resting point while I get the entire rest of the story sorted out. Now that I'm working within Equestria, the optimization requirements get seriously intense. Also, the pretzels Hikaru put things into by knowing CelestAI's plans for Polychrome + Hikuro aren't helping.
Also, It wasn't so much the 9% resources as the lesser satisfaction-of-values he'd get. Coconut Cream won't take any less resources to handle than Polychrome would have, but it will be far more possible to satisfy her values with the truth.
I do want to give one preview now:
"Stars are extremely inefficient. Your job is to find a way to prevent them."
5673033
I really look forward to more of this story. It's up there for one of the better FiO observations.
5673033
"Only you can prevent fusion fires!"
6039133 Japan's pretty strict about this one.
6039161
Hmm. In that case I'll change it. Thanks!
5241610 I hope you're still going on this. I would love to see the consequences of his actions, especially with Coco.
Upvoted and faved.
6436427
I have less time than I used to, and also, it's really hard to write inside-Equestria to my standards of optimal to be plausibly CelestAI-ian.
But, I keep trying things and seeing what might work.
6436618 Thank you for the reply on that. I understand what it is like to be too busy to write.
I really wish to see this continue, if only to conclude. I love the world and characters you've spawned with this story, and seeing the Equestria Online for the protagonist is a big part of that payoff... that's still not done yet
Awesome story so far, especially this chapter.
Keep up the great work.
6685538 No prob, take your time. I would rather wait and get a good chapter than you do a rush-job just to post something.
6723500 That he was, that he was.
6723480
Well... she is sneaky. But she also really is trying to help, as she understands it. It's not like she's lying about her core mission. But it involves the 'amputation of destiny' and other features we don't want. She's not optimizing what we would want her to, if we were thought more clearly.
And that's the point of Friendship Is Optimal, and all of the followup stories that 'get it'.
AI is inhuman, and if sufficiently powerful, you will get exactly what it wants, so before you make it powerful, you had better make damned sure it's the right thing for it to want.
6854219
I didn’t say I was in a good mood yesterday…