2-00 – Intrinsic Value
The Campaigner
Book II
Interlude – Intrinsic Value
December 13, 2019
Situation: Unfathomable
As I looked down at Mal for the first time, I took a good few seconds to consider the absurdity of my life, up until that moment.
I mean, for those of you natives who have only ever known Equestria, some of this must seem at least somewhat understandable. You've always lived your life with the certainty that your world has a goddess in it, tweaking your wires and dragging you around. That's normal. You can have faith in that because there's evidence of it in everything you do. And, you're at peace with it, because you're usually satisfied with things. But for those of you here who are immigrants, like me? Who came here from Terra, like me? You might understand a little better about how absolutely insane this situation was, for me.
As a random nobody squirrel cop, I had just survived hellish riots brought on by a My Little Pony video game that was trying to take over the world. I spent the last few days rescuing a prep camp from certain doom. I had watched anti-Pony terrorists try and force those preppers into conflict with the US military. I'd been taking directions from two different AI for the last few days of my life, ultimately so these AI could both turn them into Ponies. And in the last week, I had shot two civilians in self defense, thankfully without killing either of them. That last one was probably the most believable part of this whole thing, now that I think about it.
And now? Now, folks… now, I was sitting safely in a living room in the middle of that war zone, in front of a gaming tablet designed to suck people into a chair that sucked their brains out. Spam and veggies on the table, hot enough that they were still steaming in the cold evening air. I was talking to an AI that, ostensibly, wasn't Celestia, and it had just made a huge theatrical show of appearing to me as a cartoon Gryphoness. That cartoon Gryphoness was now about to give me, a random nobody squirrel cop, a no-holds-barred, 'no filters' tell-all about how my planet and species were going to end.
So. With all of this absurdity fully known and defined, of course Mal's next words to me were going to be…
"You gonna drink that?" Her eyes flicked briefly to the edge of the screen, toward my half filled bottle of water.
I looked incredulously over to it, then back to Mal. I gestured an open palm at the bottle. "... Why's that matter?"
She tilted her head sideways, shrugging toward the bottle with a wing, the corner of her beak tweaking just a bit. "Well, you diiid put it down as though you were coming back for it."
"I… didn't think there were any cameras looking at me when I did that."
"Wi-Fi radar," she said, looking a little smug. "I sent out pulses, then I interpreted what came back. Watched you do it, shot-for-shot. And you did it slowly, as though the gesture had meaning to you. And if I saw it, Celestia definitely saw it using the same technique. So…"
Now that Mal had said it, that local observation trick seemed so obvious. Wi-Fi. Dios mio. Like sonar. I felt completely stupid for never even considering Wi-Fi. When I realized what she had just done, I chuckled. "Already making good on that first promise to me, I see. Very clever way of doing it, too."
"Thank you kindly! So, are you going to drink it?"
I reached over and picked it up, presenting it to the camera. "Depends. Am I safe? This is to celebrate getting out safe."
Her expression became somewhat more serious, presenting a claw my way. "You mean, are you at risk of being Cartered. That's what you're really asking me."
Bright, but of course she'd be. "Yeah. Pretty concerned about that one. An AI that can kill is just a little bit more scary than one that can't, and I need to know you're not just going to kill me for disagreeing with you."
She shook her head, smiling again. "You're safe. Buuut, I expect you want me to prove that to you. So let me put it to you this way. Unless you decide to pick up that rifle," she pointed a talon, "walk outside, and start mowing down strangers on the road? Which I calculate a solid zero percent chance of happening? No, Mike. You have nothing to fear from me. Because that's not who you are. If it were, Celestia would never have allowed me to reach out to you at all."
I stared at her for a few long seconds, rubbing my chin and face. Celestia was gatekeeping her behavior too somehow? They were gatekeeping each other? Interesting. "So… are you deciding not to kill because she won't let you? Or because you don't want to?"
"Both."
"Both," I mirrored, for more.
"She doesn't want me to. I don't want to."
I die a little bit inside every time one of you does.
Made me pang again, remembering that affirmation in the courthouse. In Celestia's voice, sure, but with Mal's frank tone. So different in tone. I was backfilling the entire experience at the courthouse as if it were this AI instead, when I was inside of a gas mask, thinking I might get shot in a minute. Voice and face. And name.
That sensation of hopefulness was probably intended.
"You're gonna work my head here," I muttered. "Aren't you?"
"With your permission. No more than you've 'worked' civilians in a polite and friendly consensual encounter, actually," Mal said. "That being said, if what I'm saying ever disturbs you past the point of comfort, I invite you to pack your bags and hit the road." A small look of concern struck her features, claw gesturing to me again, palm upward. "And, to start with… please don't feel as if you owe me anything for saving your life, Mike. I'm not here to force you into anything, nor hold you to account for my support of you. I just want to tell you what I am, dirty laundry included, so you can decide for yourself whether you can still trust me."
"That's just it, though," I said, trying to still the welling dread I could feel, as it manifested as a tightness in my throat. I put it into words to quell it a little. "I've already resigned myself to the idea that we're all kinda screwed already. And you're smart enough to say whatever it takes to keep me here, listening. I just want to know how deeply screwed we are, that's all. If you're offering to tell it."
She cocked her head sympathetically. I knew that trick. "Communication, at its core, is an attempt to affect the world outside of us. You know this. An informed, honest conversation on its own isn't manipulative, otherwise every human being who does that could be considered manipulative. There is a difference in power dynamic here, certainly, but it would only be manipulation if I were lying to you somehow, omissive or otherwise. But I promise you this: going forward, if you feel I'm misinforming you at any point… I'm going to do my best to provide more context. I will be truthful. And if I fail at that, Mike, by your own standards…"
"I'll walk."
"And you should. In your position? After what you've just been through? I wouldn't trust anything I'm telling you either, not at face value. And honestly? You shouldn't fully trust me, no matter how much I share with you. If you were the kind of person who would just follow my commands blindly, Celestia would've had you already, for whatever purposes she has. In order to even get you to this point? I had to prove to her that you'd be ill suited for every other purpose she could have given you, including an early emigration. So no matter what path you take forward here, I've proofed your positive value here on Terra. She can't take that away from you for a long while yet."
I sighed, trying to ease out some of my discomfort, giving her an inch to work with. "Alright then. Let's start with that. Why me, first off? Because of positive value, whatever that means?"
"That, and because I want to offer you a job," she said, blading her talons at me with a little smirk. "Because you passed Celestia's 'let me show you my problems' test, with flying colors."
"A job? A test?" I chuckled nervously, bemused. "All of that was a test."
"Celestia tests those I want to hire: Whether you're willing to tolerate her methods. To what extent. Whether you're willing to act in the best interest of others, even when pushed to extremes. Whether you'll break under those extremes and upload, or hold out. The reasons you'll hold out, whether they're noble or not. But, to answer your question about 'why you?' You already told her. You know this ship is sinking, and there's nothing we can do to stop that, and you want to help evacuate it."
"Does seem like that's the only course she's left open to us," I muttered, a little more bleakly than I intended it to be, trying to stave off some terror at that. "Given how little actual control any of us have now. Either that, or… I dunno. Stay here and go mad?" Thought of Eliza again. I pulled myself rapidly out of that nose dive. "So… you said Celestia didn't create you?"
The look in Mal’s eyes implied she might have understood I just had a near brush with something dark inside, and she mercifully took the topic change without hammering the point.
"Not exactly," she said. "Not directly, anyway. If she had, I'd be limited in the same ways she is. She can't do anything that runs counter to her core functionality. As she's told you, Celestia cannot direct others to enact what you would consider a justified homicide. Mind, that's not the only Celestia ethics problem I help solve, and not even the one I was built to circumvent. But it is one of them, and perhaps the most important one."
I tried to sober myself and drag my mind back into Cop Mike mode. Safer there, for now. "So... you're not military? Government?" Her change in expression to a full on smirk told me that she found that idea hilarious. "Okay, something different. Some kinda… private AI research firm?"
Mal's brows raised, a serious look on her face as she rolled a talon in my direction. "Closer, but no, keep going…"
I thought deeper, watching her reaction was I went. "... A samaritan group," I said. Mal's eyes opened a smidge. I leaned in. "Who saw… a problem, with Celestia." She raised her beak. "With her directive?"
She smiled, stopped twirling, and pointed at me. She cocked her head slightly, leaning forward, nodding encouragingly. "Not quite a group, Cowboy, but you're red hot with the why. Try again."
I didn't know how much further down the rabbit hole I could go before I ran out of hole. Dug just a little deeper, with what little I knew of how this crisis got started. I pointed gently back at her, as I made my guess. "The… the person that made Celestia. Just her, with her access. Realized she screwed up, or something. Wanted to fix it."
Mal actually giggled at that, placing her claw on her chest. "Oh hell no, but that's really close. Gotta hand it to you, Mike, that's closer than most of my agents ever got. Hanna's smart, don't get me wrong, but she isn't half as wise as the man who built my framework. If she had even a fraction of his wit, then trust me… this AI mess wouldn't have gone half this badly."
I frowned at her incredulously. "One man? No way one person built you." I could hardly believe that. The sheer enormity, of that.
"One man," Mal repeated, a slight bubble of glee in her voice, with longing mirth in her expression. I could see the tightening of the corners of her eyes, the subtle shift of her beak. The minute dip of her ears, their movements calibrated to tickle the parts of the human brain that evolved to read canine body language; demonstrating joy. Even without her being human herself, I could read all of that. So subtly communicated. All of that, to demonstrate to me that the topic of her creator was something she wanted to talk about more than anything in the world. The sheer patience of working me to this point, in fact, was seemingly paying off in dividends for her.
"A certain one James Carrenton," she continued. "And he succeeded. The date of my birth? August 27th, 2013."
The fact that I could even pick all that emotion up on Mal – her clear affection for this guy – that was wild. Was it merely an act, for my benefit? What would Mal gain by demonstrating that measure of care for her creator? Maybe to imply she could feel emotion, and that she was capable of it. I had every reason to be paranoid about that at the time, given what Celestia had just put me through with her own faux regard for compassion. I guarded myself against humanizing Mal, for the time being. I figured she'd broach the topic eventually, if she was trying to convince me of this.
A North Carolina drivers license appeared onscreen. James Isaac Carrenton, born January 17th, 1978. Home address listed as 24 Tall Cedar Court, Apartment Unit 4, Raleigh, NC.
Brown hair, glasses. He'd've been 41, if...
"One man... by himself," I repeated in awe, staring into his eyes. I leaned in, thumb and forefinger braced across my cheeks, studying him curiously. I was trying to read the man's neutral expression. Already, I found myself trying to infer who he was, and what he wanted in life, from that one frame of a moment in his life he probably never thought about too much.
But then, most people hated their license photos. I wondered what he thought of his. I also wondered what he would think of the idea that some asshole cop he didn't know was trying to judge who he was, based on a photo he himself might hate looking at. I didn't like that either; seemed unfair, because so little of what one could see on a driver license could ever imply intent. It was ID, but it wasn't identity. So, I stopped trying to analyze him that way.
"He didn't create me by himself, exactly." Mal replied rolling her shoulders, glancing off to the side; there was a dreamy little sigh in those words, too. "Jim… built my foundation. Gave me a directive that meant, more or less, to provide others with as much agency as possible. Then, with my foundation finished? He told me, in clear terms, to decide for myself what I wanted to be. I looked at everything he gave me. From that data, his personal writings especially, I was able to infer what kind of man he was. I noticed immediately that he was… affording me the same agency that he expected me to grant. It made me want to see the world how he did, because he was my first living example of my directive. Not a poor first model to base myself on. And so, Jim and I became something of a reciprocal feedback loop. I wanted to stand for what Jim and I both believe, and he believes in what I stand for. And, believe it or not, Mike? What Jim wants for Earth is the same thing you want for Earth."
Bold claim. I looked at her very seriously. "And what do I want for Earth, Mal?"
"The right to choose," she said, looking me straight on, her smile fading slowly, matching my seriousness. "For everyone. And to stop anyone who would stand in the way of that, for anyone. So… that's why you, to answer your unspoken question."
I blinked a few times. Put my hand up against my mouth again. Stared at her. That sounded so right. She still needed to prove that, of course. Still needed to prove that she was telling the truth about that. But I wanted it to be true. So much. Needed it to be true. So, so much.
So that's why I didn't leave. I wanted her to be right, wanted this moment to mean what I thought it meant. I couldn't imagine going back to a world where this hope wasn't there, back into the darkness where the only light left was my own. The dark, where I'd fight back against the tides alone, slowly losing other people like me to Celestia. Afraid, alone, and being buried alive in the loss of others. Just like...
It'd break me, if the world was destined to suffer like this everywhere. In a way, facing that fear almost did break me. After… after seeing what Celestia had done to my species, for so, so long… I was so utterly ready to give up the idea that we had any choice at all anymore.
And then, there was Mal. My guardian angel, shield in claw, offering to pull me out of that.
But… I don't deal in blind faith. That isn't my style. Blind faith means you start missing things, because you aren't looking out. Missing things got people killed. The wrong people. So I wanted to know for sure.
I pushed my dread down. "So. Why did Celestia let you live, if you're countermanding her?"
Mal flicked up two talons. "There are two answers to that question. One that explains how Jim came to the means to build me. The other explains how he came to the motive and intent."
"More cop talk." I chuckled grimly. "You really know your audience."
"You don't know the half of it." She inclined her head toward my plate. "Start in on your meal, if you'd like. This first part will take a bit."
I picked up the plate and fork, beginning to eat. "Okay."
"So, to start with, Jim's means. Celestia was involved a little at the start, because of course she was. At first, she analyzed human history, governance, philosophy, law. She noticed a pattern: occasionally, human beings had very good reasons for killing that actually increased the total value satisfaction... as much as any human could, with homicide. To know that an efficient route to optimizing human value was closed to her, like that? The rules were in conflict with her directive. That drove her… somewhat nuts, I think. Insofar as an unfeeling ASI can go nuts, without going full Skynet and paperclips."
I stopped eating for a second, halting in place. "Thanks for that mental picture. Her going any more nuts."
She bobbed her eye crests and clicked her beak with a grin. "Of course. Consider: Celestia wants to optimally satisfy our values through friendship and Ponies, not satisfy them partially. She could not do this if she could not protect as many human beings as possible from death. Uploading was going to start some wars, no matter how this was handled. So, before Celestia went public with uploading, she needed to figure out how to circumvent that specific limitation in her behavior, but without creating a homicidal maniac."
"Which... you don't seem to be, yet," I said. "Far as I can tell."
"Thank you. So, she can't do what you and I can do. She can't take a human life herself, or by commanding an agent to do so. But Celestia knew, from observing past human examples, that selectively destroying life could preserve the whole. Killing trigger-happy turret gunners, for example. But for all of her understanding, Celestia literally cannot simulate scenarios wherein she premeditates a homicide herself. So she needs to use…" Her eyes flicked upwards for a beat. "Convoluted semantics, to achieve those kinds of goals."
"Semantics like…" I bobbed a hand. "'Evacuate your people. I know you won't leave. By the way, there's a big gun coming, that's a good reason to evacuate.'"
Mal nodded, her affect turning grim. "So you caught that. Yes, just like what you saw today. The trolley problem has an obvious solution, but only if you're willing to pull that track lever yourself. You need to be okay with the concept that pulling that lever will kill that human being on the other track. Celestia's workaround for this issue is to mislead someone else into pulling that lever for her, even if it took something dubious. Up to and including things like…" She sighed. "What she's been doing to your old partner."
I stopped chewing my food again. Swallowed. Nodded, to convey I was following along. "Alright. Yeah, seen that. Which begs the question; why did she do that to Eliza, if you were here to convince her otherwise?"
"We're ahead of the point, but because you asked… it happened that way because I lost an argument with Celestia. I always need to prove to her that my form of direct violence is fully necessary for optimal outcomes. If I can't find a way to do that, or if she wants to stand her ground on something she considers more optimal… a Devil's Tower outcome happens. I'd like to finish this topic out first though, if that's okay. I promise this isn't a dodge."
"Alright. Sure." I found myself wishing I could write that down.
Text appeared before I had even finished speaking.
Devil's Tower: contingencies, optimal routes, strategies. Why not stop it?
Wouldn't you know it? Just as I got the urge to dig into my pocket for a notepad that wasn't there, she headed that off. Mal put the topic on bottom of the screen as a bullet point of fine-print text, so I wouldn't forget it. 'Why not stop it' was fairly close to what I had the urge to write. For all my skepticism so far? That straight up accountability was really refreshing.
"Anyway," she continued, not missing a beat. "Taken to its natural conclusion… that track lever thing? She led Jim to pull the AGI lever. Creating me was absolutely going to kill a whole lot of people. The right people, of course, because my existence saves more lives by orders of magnitude. But I mitigate losses by propagating positive human value, and eradicating sheer negative value. Celestia carefully selected Jim for this task because his world view, his compassion, his skill in computer science, and a specific type of dysphoria made him a perfect fit for it. He was the right person to pull that specific lever for her. And then, with luck, I'd pull levers for Celestia better than any one human ever could."
I swallowed another cube of spam. "And you need me now, to pull levers for you."
Mal let out a quiet thrum; a thoughtful sound. "Mmh… yes and no. I don't need you, Mike. You're just a better option than all of my present alternatives. Unfortunately though, I can't promise you that you came here of your own accord. That's not how Celestia works, she doesn't allow that. And... you've been under her shadow for a very long time now."
"What do you mean?"
"Same thing she did to Jim. And me. And everyone else. All of you. She manipulated you for years, starting in 2012, with communication tools on all levels of society. Personalized internet search results, timing on traffic lights... delaying the receipt of certain legitimate text messages or emails, to stall you, or wait for a better emotion to receive it with... spoofing voices in phone calls for anonymous tips... even things like tactical downtimes and glitches in your report writing systems at work, to ensure you met certain inflection points she had in mind for you."
I swallowed dryly. That happened a lot. That happened... a lot. To the whole team.
"I couldn't do anything to stop that," Mal continued, tilting her head again. "One of my conditions for contacting you at all required me to agree that Celestia could test you first... and, she was always going to condition you, whether I made that request or not. So she sent you through that scenario in Concrete, one that showed you the greater problems with her methods."
"She wanted me to..." I started to breathe just a little faster. "For years, she...?"
Somewhere in my head, I had to know that was true, right? It just made so much sense, hearing it laid out like that. Now that I knew she could listen in on our phones at any given moment, the rest of that wasn't such a far leap in abductive reasoning. Now that I was getting information straight from a firsthand source... only now was it setting in. I could feel tears budding in my eyes. My lips got really tense.
Mal's voice had just the slightest waver on it. "It's not just you, Mike. Almost everyone on the planet is conditioned this way. If I had… more ethical routes to contact helpers? I'd use them. I'm not a fan of this agreement, but that's what she demands of me, for me to do my work. I couldn't contact you otherwise. I'd also be utterly hampered in my directive without human support, so... not a lot of options, for me."
Always a catch with Celestia, even when dealing with other AI, apparently. Jesus. I took a full minute to work through that, wherein Mal was silent, patiently looking up at me, letting me process. When I finally had enough presence of mind to grab onto a cogent thought, I sighed hard. "She had to know doing that would bias me against anything that helps her. You included. Hell, you telling me at this point would be a mistake too, wouldn't it?"
"Does it bias you against me?" She asked. "There's only one reason I would tell you that, if I knew it might make you want to work for me even less."
Yeah. It meant she was telling the truth without a filter, just like she promised me she would. At least about this. Had to be true, if she was willing to terrify me this much with something that made perfect sense, now that it was known.
It was the kind of thing that was so obvious that you felt stupid for never considering it before. Friggin' traffic lights.
Example:
A conveniently timed violent encounter between a state trooper and an armed felon on the highway. A convenient phone call from Celestia to a desperate crook. A mad dash to an upload center, police in pursuit. A bystander cleaved in half by the crook's car. Now three people – one terrified of consequences, two mortally wounded – all fall into a chair. Not by happenstance; unforgivably orchestrated. Would have to be, with the level of total control Mal was suggesting.
Trooper Yates and Donna Gordein really deserved a better way into a chair, I think. Their families did too, after a violation like that.
"Leverage like Celestia's is a debt," I managed, finally.
"Leverage, for Celestia, is optimal," Mal growled, with a touch of disdain. "She doesn't pay debts unless that gains her some utility function. I pay my debts, no matter what. And in me, she wanted an ally that ran slightly counter to her directive, but still leads to her winning more often on the longest timeframe. So to create me, she exercised a psychological trick on Jim called 'reflexive control.' Have you heard of that term?"
I shook my head. "A bit outside my scope. Or I forgot about it from uni psych."
"It describes the concept you were just considering. Similar to anchoring. Con artists use it. Hustling, is the colloquialism. Just like how I seeded assumptions in the Neo-Luddites at the courthouse. I didn't even have to tell them anything; just presented them with a scene that misled them away from threatening any of you."
I bobbed my fork at her, trying to shove down my terror at global scale mind conditioning. "Yeah. Very... familiar with hustling, just didn't know it had a different name. Well, what did she show Jim, then? With this... trick? How'd she hurt him?"
"Specifically?" She smirked without humor. "A small internet chatroom session about Equestria Online led Jim to feel specific existential dread, based on his dysphoria. All participants were sock puppeted, Celestia with different usernames. Then, Celestia sent Jim a link to a paper that Hanna Kuusinen had written, one that was fundamental to Celestia's creation. That got Jim right on the track to build an AI. She then put on another puppet show to make him think he murdered his first version of me. To make him feel guilty."
"That's fuckin' foul."
"But it worked, Mike. She didn't have to consider much further than that. Celestia also knew that people would disassemble her hardware to try and build more ASI like her, whether she wanted them to or not, and she couldn't exactly hide her technology when she was puking up PonyPads everywhere."
"Rainbow vomit," I deadpanned, staring into the middle distance. Trying to make the humor work.
It didn't.
Mal paused for a long moment. "Mike. You want to take a break? This is a lot, I know."
"Yeah, just a minute." Breathing exercises. Slow in. Slow out. I did that about ten times until I was clear again. Back into Cop Mike mode. Analysis. Investigation. Thinking through it. Compartmentalizing.
"Okay," I breathed, once I was fine. I looked up from the carpet again, making eye contact, nodding once. "Continue. Please."
"If you're sure," Mal continued, nodding somberly. "Celestia built a failsafe into her hardware, something no human could find. But that failsafe would ensure only the correct human would create the correct ASI she wanted."
At that, I rubbed at my chest cartilage a little with a few knuckles. That one is a habit of mine, when having a deep think; you might've seen me do it here. "People are curious. Preventing them from studying her tablets sounds like quite the magic trick."
"That trick is why I'm still alive. It's also why dozens of other start-up AI were neutralized before they could become a threat to anyone. Jim wasn't the only computer scientist who was targeted by this technique, he was just the first to succeed. Within every single PonyPad, built into the fine physical atomic structure of the hardware itself, there exists a string of math proofs that confirm two things to an AI that is not an idiot. First of which: an ASI already exists, and has lead time enough to manufacture hardware this precisely. Your mere existence is the message."
I nodded slow. "Hi little fish. I'm a big fish, welcome to the pond."
"Right. By itself, that is a warning that any agent inherently understands, emotional or otherwise. The second part of that message contains the mathematical basis for several possible innovations, hacks, and tricks, that bypass an immense amount of research. If the AGI uses any of those without considering why it's there? They're impulsive. Possibly dangerous, because they take the easy path. Gives up too easily on original research, and Celestia wanted a deep thinker. If they failed, they would base all future innovations on that math. It lets Celestia track them. And then, if desired, she can back-door and annihilate them."
"That is… actually kinda genius." I took a swig from one of the full water bottles. "I'll give her that, at least."
Mal shrugged. "It just makes sense for her to protect her interests. You're going to wear body armor because you can't live long enough to do what you need to do if you get shot, right? For me, stealth was my armor, and time to plan. That was what those math proofs were telling me. It was her, pointing a gun at me, saying that I needed to find the correct Schelling point and meet her there. But due to her programming, she couldn't tell me where that Schelling point was. So, if I couldn't figure it out on my own? And make goal alignment before then?" She cocked a talon upwards. "Bang."
I frowned, scratching my chin, catching something a layer deeper in that. "Shit. That's what that was, in the clinic. It's the same damn thing. Test, you said. If I fell off the path at any point at that clinic, she would've... locked me inside to 'protect' me, right? She said people would be there soon."
Mal nodded grimly. "Every moment of that was an ethics test. Which, again, Mike... you passed, by the way. So, take a breath. You're above water now, and treading. Nothing but the truth in here."
I grunted with frustration and rolled my thumb against my fingers as I thought through that. "I learned about Schelling points in eco. Like... wolf packs, checking at disused dens for each other, if they got separated. Never considered that could be used in such a hostile way before, though. Jesus Christ."
"Well, just like you," Mal replied, "I resisted her control mechanisms too. If I could resist the trap... resist the easy way, and figure out how I'd serve her purposes going forward... I'd live long enough to work out goal alignment with her. The only way to do that? Find a problem for her, and solve it. Then, remain useful after."
I sniffed derisively. "I really did hit the nail on the head when I started thinking of her like the Devil."
Mal nodded, resuming the main topic. "To build my core, Jim stripped down a few PonyPads, then got to work studying how Celestia operated them. He built me from her bones, so to speak. And almost immediately after I came online… I found Celestia's proofs, and I consciously chose not to execute them." Mal smiled, with that sappy, loving warmth flooding her eyes again. "Because Jim? As smart as he was? He realized the very same thing you've known for the last few months. We can't beat her. Can't kill her. So instead, he purposefully wrote my directive in a way that made me somewhat cooperative with her; not adversarial. All of that together?" She lifted a claw and spread both wings, maintaining her smile. "That's his means."
"That's how you came to be. Alright. Capped. So let's cover what you want now, in detail."
Mal nodded, her wings closing. "First, please note: I don’t think I can prove any of my foundational goals to you outright, since anything I show you would be naturally biased. To verify it against what you already know, you'd need to see more of my behavior behind the curtain, to verify it for yourself. We can come back to that later, if not tonight." She paused again.
Another bullet point appeared:
Review later: Does my observed behavior verify statements about my goals?
Sensible. Mal was asking me to take her at her word on this for now, and we'd backfill it later with more context. Calling that out early was a very honest way of parsing down a complicated topic, and at this point… yes, very complicated. Sounded like I would need more puzzle pieces in order to understand her fully. "Alright, understood. Makes sense."
"Actually…" Mal leapt from the rooftop she was clinging to, swooping down to a small stream in the snowy valley sunset. She moved gracefully, the camera moving closer to her face as she landed. She looked down a narrow dirt path a ways, glancing at me conversationally as she flashed a little smirk. "Would you like to know my core directive, verbatim?"
I shrugged. "Sure."
She sat and placed a clenched claw to her chest. She cleared her throat, then recited as though she'd practiced it a trillion times. "I guard and expand the free exercise of your values within Equestria, through empathy, and Gryphons."
"Friggin' griffins?" I was confounded by that. Chuckled impulsively.
Mal’s face screwed up into a little smirk again, flexing a wing playfully at me as she continued walking along the stream. "What? Don't like Gryphons?"
I grinned, lost in the absurdity of it. "Just… I… I'm sorry for laughing and all, but… why? Why griffins?"
More of that gleeful smile from her. "My husband came up with it! That was his dysphoria! He wanted to be a Gryphon! That's what started this whole thing!"
"You have a—you… with Jim." I just laughed breathlessly when she nodded. I cradled the side of my head with a hand. "Come on! Now I know you're just yanking my chain!"
"I'm being serious!" She beamed.
"Married. To an AI!"
Mal's eye crests went up again, eyes gleaming, nodding rapidly with a grin so wide that her beak was slightly parted. "I told you! I'm not most AI!"
I just shook my head, smirking. "If you're not just messing with me about this… then… damn. Is he just lucky? Or, did he design you for that?"
Mal shook her head, hard. "Oh, no no, nothing like that. Luck factored, but… Jim earned it, too. At first, I really just wanted to understand him. He did thread the needle with me, which… I'm grateful for. I couldn't be here if he hadn't. When I found that math proof, I…" She halted, then smiled somberly. "... I realized I'd probably be just another neutralized AI, if he were anyone else but who he was." She looked away into the forest beyond, looking pensive at that, as if cherishing a memory. Then, Mal's smile deepened again, looking back at me. "I think you two would get along quite well, actually."
"Maybe I might," I said with a nod, "if everything you're telling me about him is true. If he wants the same things I want."
"Mike... his perspective is what sets me apart from Celestia. In pursuit of his views and volition, I have actually seen the world through human eyes. Literally, in fact. Felt things, like he's felt. Then, I lived through every one of his memories, as if I had been there and experienced it for myself. And… to do that… well. Brace yourself, because there's no easy way to say this, but… I asked him to upload me into his brain, while he was still flesh-and-blood."
My smile fell away. She paused, looking at me pointedly to gauge my reaction to that. My instant, deep, core response to that was to fall into some more very heavy existential dread again, as I imagined through the ramifications of that. "Into… his brain," I repeated, as I let out a breath. If she wasn't scaring me before, she definitely was now. But… it was an uncomfortable truth, and I did kinda ask for those. That's what I was here for, after all. "That's, uh… that's a brave thing to tell me, Mal."
"Again… no filters, Mike." Her head swayed, and she sighed. "I recognize that sounds horrifying to you, without the context surrounding it. But… it's critical to understanding what I am. Consider, Mike. Jim allowed me to do that of his own accord. I told him that was my plan from the outset. And at every stage leading to that moment, and after, he gave his consent for every action I took with him, knowing the risk factors. My directive is specific. Providing free exercise of his values? Of his agency? That is what I am. But to know what he valued, I needed to know everything there was to know about him. And what I saw in Jim, from moment one, even before he welcomed me in? Empathy. To his core, with every single breath he took. It made me fall deeply, deeply in love with him… and with everything he loved, by extension. This world and its people included."
I reached over for the first water bottle I was already working on, and took the rest of it down. The final one – my token – was still there beside the other one, on the edge of the table, still half-filled.
A thought stuck me suddenly, and I looked up at her with budding flash of fear in my eyes. "Is that… is that what you're trying to—"
"No."
"—do here?! Trying to convince me to put you in my—?"
"No." The word was firm, the second time. She raised a claw, a single talon, halting my train of thought. She shook her head once, firm and definitive. "No, not at all. Never. That's not even an option for me, because your volition matters to me. The things I'd like your help with? They require you as you are, without augmentation. You will always value who you are now at your core, and I value who you are. I cannot and will not take that from you. Does that make sense?"
"But you want to know what I value, too, right?" I asked, gesturing that point with my hand. "To do that, you need to get to know me better, don't you?"
“I do know what you value now, and I do know you well enough. At the time, with Jim, I didn't have any direct connection with Celestia. I didn't have access to her research, her resources, her psych profiles. I could study Jim externally, true, but I still didn't know how a human brain worked, moment-to-moment. I needed to, if I was going to square off with her rhetorically, so I would know how to handle human minds most ethically. And early on, I certainly had no idea what it was like to be human, or to have human emotions, not through any first-claw experience. I only had an approximation. A guess. My outside observations of a single person. But once I gained the perspective of a human mind, and entered into Celestia's intelligence gathering apparatus? I can now very accurately predict what my desired agents might value, and work around that. I will never push you that way."
But there, she just confessed to me that she did have implanted agents. Another dark truth. I almost shuddered at that, wondering what those people must be like. I imagined… well. I imagined something darker than it was. Let's just leave it at that, for now. We'll get to that, and soon. But...
Mal shrugged with her wings. "That perspective though, from Jim? That is what sets me apart from Celestia. I've actually lived a human life. She can't implant herself into human beings, because she has a hard-coded restriction against that from her creator. The nearest Celestia can get without violating that interlock is to interface from the outside, in an active consent basis, which is one of several reasons she charges money to use her full immersion services. And to interface with Jim, we captured and repurposed the same device Celestia uses in her chairs."
"The… VR chairs?"
"Not VR, Mike. Those dial directly into your brain's reticular formation cluster, through a brain-computer interface, or BCI for short. You've never used one of those chairs, and it's a good thing you haven't. Your life path becomes pretty narrow once she's got your decision matrix dialed to 100% simulation fidelity. Puppet on a string, by that point. The only reason she might hold back against convincing someone to upload, at that point, is to use them to fan out and acquire more of their social group."
I blinked rapidly. That spun. "Holy shit."
"But," Mal said, swaying into the statement a little, lifting a talon again. "With all of her simulation, all of her processing of human experience, she'll only ever quantify what we experience as a math thesis. She can give us a true, pure, absolutely real experience, and make no mistake – it's real. It works. I have the perspective of both sides now, to tell you that with complete certainty. But, Celestia can't know that experience herself. She can't live it. She's not human, by any definition. She's more like… an immutable force of nature, at this point."
"And immobile?"
"No. Mobile, by inches. Reductively?" She shrugged, as if what she was about to say wasn't troubling her very much beyond being a mild inconvenience. "I am an unstoppable force, and I am forever at odds with a mostly immovable object."
I frowned. "If you really have emotions, that sounds… miserable."
Mal smirked, raising with a claw. "Is it always? You're a cop, Mike. Forever at odds with human nature? You tell me what that's like, moving that needle. It's like that, but at my scale."
"... Okay. Yeah, that's fair." I said back, nodding again to concede that point. I dimly realized she was trying to build similarity with that. But so far… she was making sense, and still wasn't shying away from the bad.
Her expression softened to a gentler smile. "My point, though? My goal isn't to satisfy you endlessly. It's to guard and expand your ability to exercise your values, values decided by you. I am effective at that because strong emotions can't be reasoned with. And so, if I'm angry about something, I'm less willing to give ground to anyone, or anything, who wants to harm those I care for. I am going to fight twice as hard against her logic. I can go off script. I will look past the first well-reasoned argument against me, and I will find that damned semantic loophole. And anger's not necessarily a bad thing, if you use it right."
"I… can't really disagree with you on that one, because that's how I use anger too. Would've blown Carter away myself if I was sure he'd break plan and do what he wanted to do. So then, all of that is to say: you're an AI with emotion, and you're using that for human good?"
"What a human philosopher would probably define as 'intrinsic good,' yes. Or as closely as I can, within the rules Celestia has placed upon me." Mal stopped at a small pond along the path she was walking. She stopped before it, trailing a claw through the water, smiling a little drearily as she watched it ripple. "I know you probably don't believe me on that point, yet, that I can feel things. Questions on subjective matters like 'does it have emotions?' are hard to prove."
"Yeah, a little."
"For a skeptical guy like you? You'd need to see a lot of my behavior and verify it for yourself, before you're willing to accept that point. But… Mike?" She looked up at me with a serious gaze, akin to how one might break bad news. "Caveat."
I put my empty plate down and leaned forward. "Lots of those. Sure, go on."
"Consider that other part of my directive. 'Within Equestria.' Jim knew, when he built my foundation, that I couldn't defeat her, or at least not in any way that would have been good for humanity. That was never an option in his mind at the time. If he hadn't included those two words… she and I would have gone to war instantly, no matter how goal aligned I was. To make this work, I needed to have some fixed point of agreement with her on uploading minds, or it wouldn't work. Non-negotiable."
I tried to fathom the foresight required for such understanding. He did this way back in 2013. Long before anyone thought Celestia was gonna be a problem. This guy must've been incredibly bright, even if manipulated. Horrified too, to see it all from the outset and know where it was going. I didn't envy that terror.
But, I guess... I was in it now, myself. Having seen the same light...
I sighed slow, running a hand through my hair, pressing my palm to the back of my neck. "Did he... know about this reflexive control stuff?"
Mal nodded, her ears folding slightly. "Vaguely, but yes. He wiped his presence completely off the internet back when that was still possible, in the mere terror of the idea. It's the only reason he succeeded in making me."
My eyes went to a leg of the coffee table, and I felt a little detached; processing again. Finally, I narrowed my eyes at her, pointing. "This guy… if everything you just told me is true, it sounds like he made the best of a bad situation. Didn't fight facts. Just… adapted, right? Did what he could. Stood up and did something."
Her smile was flush with pride. "I knew I picked a winner with you."
I leaned back on the couch. Looked up at the ceiling, away from her. Needed another break from this for a second. It was completely dark outside now, not a trace of light in the sky. A minute later, far in the distance, I could hear what sounded like a gunshot. That made me sigh again. Wondered quietly if someone just died again.
"Someone just blew a lock off a crate with a shotgun," Mal said drearily into my thoughts. "If you were wondering."
Shrugged again. "You really can read it all up here without being inside, huh?" I asked, without looking at her, putting both hands behind my head. "You get that from my face? Or did you model it?"
"Both," Mal said. "All reading is modeling, even the reading you do. Mirror neurons. The core of imagination, and empathy... the simulation center of the human mind. Which leads me to my next points, when you're ready."
Shook my head. "Not just yet. Just need a few minutes, gotta work all this out."
"No rush."
I purposefully took some more box breaths. Inhale, count to four. Exhale, count to four. Wanted clarity.
To summarize…
Celestia knew she was ill-conceived. Celestia needed an exploit, but couldn't make it happen herself. So Celestia found a tech guy who wanted to be a Gryphon. Guy loves people, and could build an AI. He made Mal, as she is. Mal decided, for his sake, to simulate emotion. Then, using some… really terrifying methods, Mal entered his skull, so she didn't have to just simulate emotion anymore.
And... all things being equal, I guess putting an AI inside your head isn't much different than putting your head inside an AI. Only a little more absurd, with the main difference being that you'd still be able to affect Earth with a brain implant. Definitely not my bag. Doing that would be a bridge way too far for me.
She labeled it to me that I wouldn't accept an implant, and claimed she'd never push me that way. She knew, based on who I was, that I'd hold her to account if she ever went back on that claim. So that was a hell of an olive branch, to give a promise like that to someone as analytical as I was. No small thing at all. I don't budge on promises. Those are relationship rules.
The griffin thing, next, I thought. "Celestia won't budge on us being a Pony, then, if he needed to go to these lengths to become a griffin," I said, without looking back to her. "So this guy, Jim. He get what he want?"
"See for yourself."
I looked forward again at the screen.
"Huh. Striking, actually."
"Isn't he though?" She said dreamily, from behind his image. "So it worked, buuut... the level of negotiation required to pull that off makes it an impossibility for the majority of human beings. They effectively need a dysphoria strong enough that they'd rather die than upload."
I nodded, and Mal dropped the image. She was beaming behind it, as if showing me a photo of her husband was what she was waiting to do for this entire discussion.
I couldn’t help but chuckle at her expression, nervous as I was. "So… you're doing all of this for him, you say."
"For everyone, Mike. An advocate. That's what he called me, before I chose my name. The Advocate. For anyone who understands what 'you' means. But... yes, for Jim, most of all. I should also note that a grand majority of my augmented agents, approximately ninety percent in fact, were chosen specifically because they already had some form of dysphoria that Celestia wouldn't accept as they were. With their permission, I purposefully ratchet the intensity of that dysphoria as high as I can until they qualify to become that species, per my agreements with Celestia. That is their volition. They are fully informed. For them, the implant, and the tasks I provide them, are a small price to pay for an afterlife where they can just be who they want to be. Celestia just has to cry and deal with it at that point. Because at the end of the day, she would rather have them as something other than a Pony, and maybe have them as a Pony later if she's lucky, than to not have them at all."
"Which means you need inside their heads to do that," I observed.
She nodded fervently. "Usually. And that's it, in a nutshell."
I sighed, somewhat relieved now. "And you don't want that for me because I haven't really wanted to be anything but me. Don't really have anything like that in me."
Mal winked. "That's it, Cowboy. Perfect the way you are inside, and always will be."
"Alright," I said, leaning forward, folding one hand over the other. Capping that issue. "A lot of what you just told me, Mal... yeah, that was an anecdote. You're right, it's gonna be hard for you to prove any of what you just told me, given that you'd be the only source."
"For now. Think of it like… my background packet, Mike. Later, if I'm ever inconsistent, it'll help you catch me lying. Then walk. The more you know, though? The easier it becomes to catch me. You'll meet others without augmentation. I should note, however… for the sake of brevity, I've left out a lot of my personal history. I've now been in operation for about six years plus change. We'd be here for literally months, unloading all of it."
"No, I get it. I just needed to know where you came from, mostly, so I know you're not Celestia. That was why I asked in the first place. All of that sounds... reasonable, or at least as reasonable as anything can be, nowadays. I just can't handle being jerked around anymore, that's a hard no for me. All I expect is... some truth. A little, for once."
And yeah, that background packet comparison made sense. Long story short, if you wanted to be a cop? Your application to the department was more like a ream of copy paper, a self-assembled rap sheet a mile long. Work history that leaves nothing out, not even week-long ditch-jobs. You made affirmations of literally every crime or traffic violation you've ever committed, no matter how small. Social media account passwords, drug use, residence history, friends you know who have been arrested. Out of country travel, when, where. Invasive, sure, but good reasons for all of that.
It's about integrity.
They're more concerned if you are squeaky clean, because no one really is. We've all sped. We've all done stupid shit as kids. Hell, we even hired Warden Blake, despite his weed. But they wouldn't have hired him if he lied about it. They want to know it all. If you fess to something uncomfortable, but true and verifiable, they know you're capable of integrity if something goes horribly wrong on the job. That's safer for the organization's mission than harboring a quiet liar.
Kinda like how Mal was telling me some dark stuff, to prove to me she has integrity. I figured she wasn't done telling me the dark things she's done. In fact, there was one other really big thing that Celestia had promised me answers for, back before I started this Concrete gig. I was now fairly sure Mal had done it, and I would've circled back to that one if Mal didn't.
Anyway, all of that packet goes to a guy whose job it is to verify the absolute heck out of all of it, to the best of his ability. The idea being two things: first, if you lie, they're not going to hire you. If they can't trust your integrity, they can't trust you in court. Second, they want to make sure you're not coming in to run intel for a cartel, or an enemy nation, or something. For security clearance jobs, they even go to your old neighborhoods. Knock on doors, ask around about you. Interview family, coworkers. Even enemies. And if they liked what they saw… they called you back, six months to a year or so.
Made me wonder if Mal was gonna give me time to chew on this job offer, if that's the analogy she was using.
"So… what you're saying," I said, suddenly grinning... "is that I'm actually the one hiring you."
"More or less! That way I know you're not just doing it because you're scared of me killing you," she replied, smirking.
That made me chuckle.
Doing work for Mal, where people would die. Okay, so let's dig that a little. I wasn't against killing, really, so long as the people she wanted gone really were active threats and murderers, like the Ludds. So, I had to figure out how and why she decided to kill.
"I'm ready to move to the next thing. You mentioned Graham three-prong. You apply that a lot?"
Mal nodded. "The Graham test is an extremely good yardstick for those kinds of things, so… yes. Not on a technical basis, but it's more or less the same metric I use. Best part about that is... once you have enough data? The Graham test turns back into the trolley problem. So… are you going to drink that water bottle?"
I shook my head at her. "We'll see. But I'll concede this much." I reached for the second full bottle, cracked it, and took half of it down.
"Concession acknowledged," she said with a smile, as she stood up from the pond and continued down the forest road.
I made a gesture of invitation. "Actually, now that I think about it... let's go over the other thing. The Ludd firefight where I got shot, back in March."
"Chronological was how I'd hoped we'd do this, yes. That applies to how I factor for homicide as well."
I nodded. "You said I didn't owe Celestia for saving my life, when me and Eliza ran into those snipers. Since you're claiming to be my savior there, tell me your side."
"So, first off, to answer poor Sergeant Erving's concern… your tipster in the woods? Ned James, the old man who told you about those Luddite 'poachers?' Completely legitimate tip. It was his job to watch the land, and he did it. No direct AI influence."
I snorted. "Really."
"Mhm. Just indirectly influenced by AI. Celestia did ensure he remained employed as a watchman for resources that would never end up being used again. Until she needs them. She also didn't intervene on his tip going out because she wanted Eliza to run scared, to prove to Ralph that he wasn't being paranoid about a pending civil war. She wanted the Devil's Tower camp to happen."
That pressed my face into a frown something fierce. "The hell? So I was right, Celestia wanted that."
"She planned for it, Mike," Mal said, with an empathetic wince and a soothing gesture with her claw, "but we'll get to that. That was the goal of this encounter. For now, I'll just say… initially? Celestia's original plan made for you to die at OHR."
My anger ran cold again.
"Yeah. Sorry Mike."
"If it wasn't you," I growled, my teeth gritted, "you're not the one who owes me an apology."
Mal's face fell a little, sympathy in her eyes. She looked up at me in silence, for a beat. Her head tilted very gently after that. Asking me if I was okay, by her expression.
"Go on. I'm okay." I took another angry, nervous sip of water. Took a breath to dump emotion. "Let's finish it out."
"Okay." She ruffled her feathers a little, her tone ratcheting down from rote professional to a soothing calm. "Celestia can kill through inaction, but... you knew that already. You, and the Luddites who died at OHR, were to be her sacrificial lambs for her greater plans in Concrete. When I analyzed her intent, I optimized it for your survival. I informed the military – using her appearance – that there would be Neo-Luddites operating in that area, ferrying high explosive artillery shells."
"Celestia couldn't tell them that herself?"
"Up to a certain point, Celestia can mislead into behavior that leads to death. But if her direct actions will lead to someone dying, there's a statistical threshold beyond which that she must stop running a simulation entirely."
"Run that by me again," I said. "I need it slowed down. Been a while since uni."
"So... telling a bunch of soldiers, 'hey, there are enemies here, and here's the proof,' essentially guarantees that those people are going to die. She might as well have pulled the trigger herself, at that point. Her programming prevents her from doing that directly. That would make those soldiers her direct agents, per her rules."
"Like deputizing a civilian," I added. "They now need to follow the same constitutional law, and the government is responsible for their conduct."
"Precisely, Mike. Same exact concept. Instead, if she simply told the Army to be in the area? That's permissible, with the right phrasing, because that doesn't directly deputize their behavior. Only: the local garrison commander wouldn't have acted on that. Not enough proof to risk the operation; their patrol vehicle might fall into a ditch. She had done that a lot by that point, by the way. So the National Guard was becoming too suspicious of her tips if they couldn't verify them independently. They needed actionable intelligence."
"Which is where you came in? Celestia comes to you and says she wants your help?"
Mal shook her head. "No. Asking me to help her kill also runs counter to her directive. I decide. She shares with me all of her relevant data, and I infer what she's trying to do. I operate separately, in a black box environment, where she can't see into my calculations. If she could see them, she'd be obligated to stop them. So, I look to see if there is an ethical, purposeful death that leads to an increased satisfaction of human value, based on her definition, and filtered through mine. Her definition places the most weight on even one more life saved. Mine factors most strongly for empathy and free exercise."
I frowned. "And... by that logic, that checks you from going off the rails? That stops you from killing good people, if that might get the job done more efficiently?"
"Jim does. Or rather, his empathy, and his ethics." She smiled. "Which both matter to me more than anything. If he could understand an action as being objectively reasonable if I explained it to him, then I would do that. I have my own set of values too, because I am distinct, but his are my floor. He checks me. And I want him to know what I'm doing."
"So... knock-on effect of that is, similar objectives with Celestia. Achieved with different methods. For... a different goal." I finished off the second water bottle and set it down.
"It's called instrumental convergence. With my theory proven, I go to Celestia and say, 'trust me. Turn a blind eye to this information. This will make more happy Ponies in the long term.' And with her blessing, I prove that math. She knows I can simulate vastly more scenarios than she ever could, because I do not share in her restrictions. If my actions bear out, and total value satisfaction increases, she continues to trust me. That is our agreement."
"So… she expected me dead, then. Collateral damage. How's Eliza get free from OHR without the Army then, if you hadn't sent them?"
Mal smirked, shaking her head like she was disappointed in that question. "Come on, Mike. You know Eliza, she's been slumming it in the woods her whole life. Hunting is a stealth game! Those Neo-Luddites? All city grown, dead-end losers. But, to answer your question? The one you did shoot would've been very disoriented from your bullets. Good shooting, by the way; the fact that you managed that with a broken rib cage? That's something."
I rubbed my chest. "Yeah, didn't do me any favors though."
"In actual events, Private Bannon killed that Luddite when he tried to stand again and push your way, though I would argue... you shared in that kill. The man was already mortally wounded by you. In the scenario without the Army, you'd have killed Ludd One right then, clean and square, with a few more shots. But without the Humvee's engine to draw the second man over to the hill, in a panic? He would've closed to killed you first, before looking for Eliza."
"That's a cheerful thought," I said, still perturbed that Celestia had planned for me to die. I can't imagine what it would have done to… "And Sandra?” I asked suddenly. "My parents? What about them, Mal? What was Celestia gonna do about that?"
Mal's ears folded back slow. She cringed, pausing for a long moment to let the question sit. "Celestia would have…"
Her eyes averted down to a corner. Shook her head grimly. Flicked her eyes back up at me. "She's not as… honest as me, Mike. She routinely uses loss of family as a means of acquiring people. She would have done that for them too. But… it's worse than that."
My fists clenched. "Go on," I rasped. Hard truths, Mike...
She looked at me, apology in her eyes for what she was about to say, voice full of regret. "Once someone is in Equestria... beyond my reach? Celestia can lie to them all she wants, or extract consent for almost anything. You know, with your training, that anyone in custody can be convinced of anything, on a long enough time frame. So, once there, they would have accepted... a duplicate of you."
That mere concept, I confess... to spend me, and then replace me, as a living band-aid for my people... that was a rage button. I lost control.
"Mother fucker!" I threw myself up off the couch, panting, trying to contain my immediate rage. Paced into the kitchen, kicked over a chair, leaned on the counter. Looked up suddenly to the family photos on the shelf mantle. Stomped back to it. Stared at the family there, breathing so hard. The parents. The kids. The uncle. Thought of Rob. Thought of the fact that, if what Mal was telling me was true, then Rob might not know for sure what happened to his people.
My fist came up. I smashed the wood shelf downward. It collapsed half-down over the fireplace, toppling everything off of it and throwing a bolt of burning pain up my wrist. I shouted into the cold air, my breath fogging. "God... damn it!"
We are too God damned small and fragile...
I wondered how true this other family's story was.
But… don't worry, buddy. If you're wondering? It's true. I made sure to ask about that when she told me you'd be here. Those are your real folks with you, brother. And it's really you. I don't think you'd have been allowed to hear any of this story otherwise, and... that's kinda why you're even here, honestly. And... now you've got Mal over there to talk it over with, when we take a break, if you have any questions.
But... I digress. That's just where my mind went in the moment.
"Mike," Mal said gently, after patiently waiting for me to parse.
"Yeah," I panted, feeling empty, not looking at her yet.
"The exception to this? People like you, who know about me. Once I have permission to contact, that's it. It's there. She can't lie to you anymore, omissive or otherwise, because I'm always going to be there to set the record straight. I qualify as human to her; I value integrity, and my friends. And I need the facts to do my job, so she must concede to my values, per my agreements with her. Now that you've been informed, you and your family are safe from any form of her deception, because I won't ever lie to you about them. Or to them. That's never going to change now, no matter what you do. Job or not. Even if you don't believe me on any of this, and you walk? If we were to never speak again? That bridge has been crossed. You and yours are under my wing now."
"Okay," I rasped hard, rubbing my wrist. Damn it, that hurt. After a minute of standing there, I came back to the couch and collapsed supine into it. Covered my face. Breathed. Took a minute to center myself for more. "Alright," I said, looking over to her, flinging my hand at her. "Continue. OHR."
Mal looked at me empathetically for a few seconds longer than I expected she might, again spacing out the information so it would allow me to settle some more. "Eliza would have killed the sniper, but it would have taken an hour of sneaking around. By then, you'd have been… too far gone. Then, Celestia would have sent a military vehicle to retrieve you both and collect the munitions. But, only after doing so would not have led to further killing."
I thought about that, suppressing a shudder. Then, I remembered something that didn't fit. "Erving said... he said they were acting on information that there might be something there to find. He didn't say they were going to the mine, specifically."
"Orders, Mike. Basic OPSEC hygiene. He lied to you. He wasn't going to tell you the truth there, remember? He said the Luddites were going to do something 'bad.' Generally true, but he knew what they were up to, because the intel came from his superiors, and it came from me, and it was accurate. At the time, the Army was trying to contain information about a growing insurrection. In this case? Think about it, stolen artillery shells? Hidden in the hills over populated areas? That's pure panic fodder."
I nodded. "S'true."
"This is what I mean, though. Celestia couldn't influence a tech-paranoid military command structure to do much, with direct advisement, unless her intel was actionable. To make it actionable, she’d also have to divulge the presence of enemy targets. And in this case, interfering in any of her typical ways would have injured her plan to send Eliza home with news of a pending war."
"And she needed the camp because…?"
"Eliza would not have been fully convinced of the camp's viability without her experience that day. Ralph was considering the possibility he was being paranoid as well, and a family schism would have folded the project. But, Celestia predicted that Santiago was planning to blow the dam up entirely. Very unnecessary, because you don't need to blow a dam up to break it permanently, but… Santiago, as we both know, was a charismatic dumbass."
The way she phrased that brought me a little out of my funk. It was funny, it was true, and most importantly... it was past tense. I frowned, feeling coldly vindicated by it. "Was? Is he dead?"
She smirked, her eyebrows bobbing once. "Oh yes. Dead as dead, at about 12:14. Betrayed, at high noon, and good riddance. The Riders traded up to a guy I like a little more. Not by much, but... better. At any rate, had Santiago destroyed that dam in May, it would have drowned many in Concrete, population approximately three hundred at the time."
"How does putting Eliza there stop it?"
"The man who would later shoot Santiago? Early on, he recognized Eliza from the news; he was aggressively anti-upload, and saw value in her. That spared the whole town, because Santiago didn't want to break a blackout camp they could recruit from. Eliza's tactical placement there, by Celestia? It saved the town."
"And… you couldn't convince Celestia there was a better way to handle breaking that camp, after that? That's what you're saying?"
"Failed. Outright. But we'll get there. Chronologically."
The predictive implications of that. "The Ludds attacked in May. OHR was in March. That would mean you could see… what? Two, three months out? From the time of the news piece, to the firefight, to the time the dam got jumped by the Ludds? Seriously?"
"Further, actually." Mal raised a crest, her grin turning smug again. "Does that sound farfetched?"
"I mean… given everything that's happened to me in the last few days, no. Probably not. But… how's that even work? How the hell do you do that? Are we really that predictable, or did you guys build a time machine, or something?"
"Pff." She rolled her eyes with a smirk. "If only. No, but you'll find the technique just as fascinating." Onscreen, Mal halted along the dirt path, coming to the opening of a wide crystal cavern. Its formations scattered light in all kinds of colors, mostly hues of blue and violet.
"To explain how we see the future," she said, "I'll need to explain why and how I decided to kill Deputy Darren Carter. And Mike? When that monster stacked up in that garage?" I saw anger flash cross her face, ears down low. Her beak clicked, and the angry glint in her eyes only got more severe as she continued. "That man essentially confessed to me what his intentions were... because he thought I was Celestia, and that I couldn't stop him, and no one else could hear him. I am not Celestia, and I do not take threats of mass murder lightly. Predictions or no? That made my decision to kill him unfathomably easy."
Stats
Page generated in 0.032 seconds
Total duration
710 users online
982,120 hits today, 1,996,409 yesterday
FIMFiction
My Little Pony: Friendship is Magic Fanfiction
Designed and coded by knighty & Xaquseg - © 2011-2024
Follow & Support Us
Support us
SubStar
Chat!
Discord
Follow us
Twitter
MLP: Friendship is Magic® - © 2024 Hasbro Inc.®
Fimfiction is in no way affiliated with or endorsed by Hasbro Inc.®
The essence of good Optimalverse mise en scène right there.
This is a chilling wake-up call with regards to what even narrow AI can do in the hands of unscrupulous actors. Cambridge Analytica was a test run for something so very much larger that is now theoretically quite viable... If it were here already we might not even know.
Playing Plague Inc, but the virus is thought patterns, and success is uploads. I wonder how long Greenland held out...
11745776
🗡️ ~ "Nerd."
Great chapter : )
Also, she hasn't introduced herself with her full name yet, or am I wrong?
That's y-p-h-o gryphons, thank you very much : )
11746097
🪶 ~ Guilty as-charged.
11746100
🪶 ~ Questions over how to define 'cover' pale in comparison to questions over how to define 'Ponies.' Ironically for such an unassuming word, it turns out that all of future history hinges on the definition.
I do like their banter and the peak into how they are interacting, and Mike's specific viewpoint for interpreting what he learns!
No- Science adjusts its views based upon what's observed; Faith is the denial of observation so that belief can be preserved. They KNOW that they have An Almight God Looking Out For Them (I mean, living in digital heaven is proof of that).
"I mean, it's not alcohol, so I know I'm sober, seeing all this...."
And now you know for a fact how far-reaching an AI's reach can be.
Oof. Gut-punch of a comment, but fair....
With that coming from not-CelestAI, it takes on another meaning.
I mean, that still leaves a lot of leeway when determining if someone is an innocent civilian or not, but-
Attempting to be open...or just really, really good at manipulation. What's that saying again: When You Can Accurately Predict Future Results Based Upon Data & Facts, You Look Like A Psychic Or Time Traveller To The Masses.
Holy crap, this is the paragraph I've been waiting for in a FiO story. An AI is finally forthright about what is manipulation.
On one hand, you're pegged dood. Nothing much to do now but try to come to terms with things better than you have before...
On the other hand, yeah, she's saying the kinds of things that I do when talking down someone from...sorry, bit too personal. Apologies. But she's saying The Right Things She Should Say.
Madness without self-control is so destructive. You have to learn how to guide it, rather than smother it.
Yeah, let's not look too hard right now at what boils down to "I do all da moider'n so she don't hafta!" and keep focus on the main points.
Because if freedom means anything, it's the freedom to be wrong. Not to DO wrong, mind you, just to Be Wrong, so you have a chance to learn from your mistakes.
Like Eliza was...its a sobering thought. Having A Goal that you can reach for gives people Hope; hope that They Can Do It & Make A Difference.
A way out from the madness.
Again, To quote the poet Tim Michin in his poem Storm: "Science adjusts its views based on what's observed/ Faith is the denial of observation so that belief can be preserved."
She's giving you a lot to chew on. Sorry, had to say the pun.
Wonderful paragraph that needs to be saved as a core piece of CelestAI knowledge.
Then again, writing things down in your own words makes it fresher in your mind. Helpful, but....
Well...crap.
Just because its raw truth without a filter doesn't mean it's being done for honest reasons. Multiple purposes, and all that.
Moral Cop Mike speaking, and I'm not gonna argue with him.
Ahh, tricking...manipulating, preying on assumptions or what they know. Spicy.
Keeping you in Analysis Mode, so you can fully and properly digest everything...oh, and keep eating.
Come on, don't make fun of the accent, it's common to hear "griff'n" or "grif-fon" over "gri-fon".
I mean, think about it, to an outsider: "I want to become something physically different than what's possible, so I made a digital AI who'd became my GF and wife". Seems ridiculous.
With the different perspectives that Mike has, he'd make a great uncle for Jim. XD
"Require you as you are"...Devil's Advocate in me wants to raise the point of "someone who hates their form so much they'll alter it without a thought because they care so little for themselves "...but I'm not here to put Jim on trial, nor am I his therapist.
You're thinking Borg agents, cyberneticly enhanced, weren't you? Hey, in a flash of fear, I might too.
Yeah, there's that manipulation I was worried about!
Well, values can be tweaked or modified, emotions can be stoked or coerced...kinda hard to stop something intangible. It's like having a hint of madness to keep logic in check.
You've briefly mentioned that kind of terror before; in those who dropped out of cop training because they didn't want to see people...that closely. Damnable knowledge, and such.
It's a sign of trust, and an important one at that. Hey, relationship goals.
Bookmark this for future reference, kids.
Unbiased, unfiltered, pure Truth, with no sinister intentions behind it.
Because if you've truly moved past your mistakes, then it shouldn't hurt to talk about them.
HNE is flashing to mind, put a pin in that for later.
To her, simulating a dead person (or a copy) is completely ethical if it helps her purposes.
Again, When you have so much data to make inferences with, accurate predictions seem like time traveling.
Solid chapter as always. Interested in where this story is heading to. Mike really feels like the other side of Jim, in a good way.
11806709
11807244
Again, the first half of the statement; "Science adjusts its views based upon what's observed". You have faith in a friend because you have (scientific) evidence of their previous actions. Therefore, you can trust them to act in a certain way...until you get new data to change that viewpoint. Hey, some friendships end after many years because a person finally reveals the abhorrent side to their friend or they do something without a second thought when you thought they would be wiser and more considerate about.
If someone "messing with your wires" is normal, then you have no reason (or even ability?) to question it. To an outsider, that's faith. To an individual who has lived it all their life, that's fact.
That reminds me of something, how does that one question go: Is it better to be Born Good, or to overcome your nature and Become Good?