//------------------------------//
// 3 - On System Failure:  Restart Gracefully 
// Story: The Advocate 
// by Guardian_Gryphon 
//------------------------------//

“I think it's fair to say that personal computers have become the most empowering tool we've ever created. They're tools of communication, they're tools of creativity, and they can be shaped by their user.”
- Bill Gates

“You can make anything by writing.”
—C.S. Lewis


December 25th 2012

I went home for Christmas.  The family farm, not my apartment in the Triangle.

Though I could not bring myself to abandon the entirety of my concern about Celestia, and EQO, nor the majority of my preferred precautions...  I wanted to escape thinking about them for a while.

What I needed was a cold, clear dose of perspective.  Hugs too.  Comfort food.  A chance to reset and center myself.  I was exhausted, strung out, traumatized.  Head spinning.  Stomach crying out for something besides processed noodles.  Mind and spirit screaming out for more than two consecutive nights of more than five hours' sleep.

What little was left of GryphGear v1 I had taken directly to the county landfill.  There I'd permanently interned it in a trash smasher.  Watched very closely as the entire assembly was shredded and minced to a pulp by carbon-steel teeth.

I worked the rest of December like a man in a waking coma.  I didn't say a word to anyone beyond what was strictly necessary.  I 'slept' each night...  But it was barely sleep.  More of a feverish tossing and turning in the throes of tortured half-consciousness.

And at last, Christmas rolled around.

I packed my remaining three PonyPads at the bottom of my Faraday box, threw two weeks of clothes into a duffle, gassed up the station wagon, and drove home to South Carolina.

Driving, to the sound of an eclectic mix of synth, country, and soundtrack music on random shuffle was a salve to the raw wound on my mind.  The city fell away to highway, and then by and by eight lane divided highway became four lanes, four became two, and two lane highway gave out entirely to back country roads.

I'd grown up in dairy farming country.  Rolling gentle green fields full of speckled cows.  Clear hot blue sky.  Stands of pine and blocks of corn and cane fields for silage were the only thing besides houses, or the occasional church, to break up the vaguely ordered, but still slightly disorganized rambling pastoral scenes.

I hadn't fit in especially well culturally...  Most people who were born there stayed there.  Worked the farms, and by and by inherited them, then passed them on to their kids.  Sometimes I wished that had been my choice instead of a degree program.  

But I was a born nerd.  Cursed to be an engineer, like the trash man in the Dilbert skit foretold.  I'd seen too much Star Trek too early in life to be content with being a farmer.  But I was starting to finally really understand how one could be content with that path as well.

Some aspects of going back were hard.  I was used to hearing six languages a day at lunch during Uni, and two or three a week at SAS.  When I wasn't ramen binging, and was on a healthier daily routine, I'd regularly eat a varied diet from a panoply of cultures.

By contrast my hometown had more cows in it than people.  By two factors of ten, probably.  More than half the inhabitants had never traveled more than two hours from home.  99.9% were Caucasian like me.  That created a sort of bubble of disconnection from the outside world.  

It was sometimes hard to reconcile how much I loved those people, and how good I knew they were, with some of the more...  Difficult opinions and beliefs they held.  Mainly because they'd been raised into them for generations through a complex mélange of implicit biases, intentional cultural choices, and accidental self-reinforcing echo chamber loops.

Bias is subtle.  Insidious.  Horrifically dangerous.  And bias is something we pass on.  To those around us, and to our programs.  I'd known someone at Uni who'd made that her entire graduate thesis.  She was studying the nascent field of crime prediction, and the ways that old bigotry and unintentional hind-brain implicit biases could creep from developer to algorithm.  And then become self-reinforcing.

I was freshly haunted by how close I'd come to letting my biases get the better of me.

There were far more good things about going home than bad though.  I adored the food.  Southern cooking is medicine for your soul.

Loved the scenery.  Loved the reams of adoptive grandparents I'd grown up with via the church family there.  Loved seeing Mom, and Dad, and the farm again.

A lot of my peers at Uni, and later my coworkers, had made a lot of assumptions about me when they learned where I grew up.  More implicit bias.  No one is immune.

That had taught me, gradually by and by, that a lot of people were prone to throwing out the baby with the bathwater.  They saw the biases in others, and instead of being conscious of their own, or looking for ways to mutually adapt and grow, they wanted to throw those people out, along with aspects of their culture which were in no way inherently wrong.

Aspects which were often times, in my opinion, inherently *right.*

I'd come to feel that growth was a trap.  Most technology too.  Strange thing for a programmer to say at first blush, but if you think deeper, it starts to make more sense.  I'd started to see the technology-driven feedback loops of the world.  The downward spirals, everywhere.  The ways we'd already enslaved ourselves to algorithms that demanded hateful polarizing content, ever-growing profits at any expense, and the 'optimization' of monetary functions above all else.

Some of the main things country folk seem to know, that are right and true;  People are everything, and technology is no substitute for love, honesty, and hard work.  Applejack would have been right at home on my own family farm.  Maybe that's why I related to her so much.

As I pulled into the gravel parking area beside the farmhouse, I threw the car in 'park,' cut the engine, and spent a long moment just staring.  First at the house, then the couple of barns of varying age and construction, and then out to the fields beyond, and the treeline beyond that still.





In that moment, it felt like everything was an intractable mess of contradictions.  

Laying aside the wider realm of philosophical questions regarding AI and transhumanist speculation, I was still left with a personal world of tangled emotions and ideas.

Gryphons were not just a part of my identity because I loved them, in the way that I saw them.  They were a part of my identity because I had shaped, in my view of them, an ideal of the person I wanted to be.

The physical idea was important;  Don't get me wrong.  I would have traded any amount of personal pain, suffering, and effort to have wings, and a beak, and claws, and a tail.  A fact I was very ungenerous with in conversation, even with those closest to me.

But there was much more to it than just the physical.

Seamlessly ingrained and intertwined with the whole thing was an ideal.  That's what had ruffled my feathers so badly about 'Griffon the Brush Off.'  The depiction ran counter to my ideals in such a clearly negatively-coded way.

I wanted to be kind, selfless, strong in defense of others, and brave.  In setting that goal for myself, I had made that vision of Gryphon-kind a symbol, and that symbolism an integral part of what the word 'Gryphon' even meant to me.

And, too, wanting to be a Gryphon was recursively part of being one.  To me, a Gryphon was something that was wholly comfortable in its feathers.  Never wanted to be something else, or felt conflict about what it was.

But that wanting...  That longing...  Which was such an important part of the concept...  It had come into direct contradiction of my other ideals and goals.

Then there were the contradictions in the world around me.  The gnawing sense that the whole entire Human species was becoming fractured and disconnected the more we immersed ourselves into technologies meant to bring us closer.

The dull, insistent, roar-like mental sound of the quiet knowing...  Knowing that even if Celestia was not everything I feared she was, that one day soon someone was going to make the same mistakes I had.  Only there wouldn't be a handy cup of coffee, and the routers might be on WEP instead of WPA2.

And paired with that bruise-like ache of knowing, a companion thread of nihilism.  The feeling that if we didn't invent something smart enough to take over from us, that we'd just sear ourselves to death in a world plundered of all its resources, to the point of uninhabitability.

We, the collective of Humanity, were stuck on the horns of a dilemma.

And individually, so was I.  The realization wasn't new, but phrasing it that way to myself mentally was.

'What are you going to do?'

That was the first question.  But it was tightly entangled with the second, and perhaps vastly more important one.  The one so few people ever consciously answer.

'Who are you going to be?'

I knew if I stayed on my old course that I was going to turn out to be someone I didn't like the look of in the mirror very much.  Yet I could not see a path forward that allowed me to become the person I wanted to be, while still fighting to carve out the outcomes I wanted for the end of the world...

The outcomes *I* wanted...

I drummed my fingers on the steering wheel, and closed my eyes.  There was something in that...  A tiny peek at the buried form of an answer to a question I still couldn't phrase properly...

The staccato sound of a *knock*knock*knock* on my window made me jump so hard I hit my head on the roof of the station wagon.

My head snapped left to see Mom's smiling face.  She was always doing that to me as a kid.  I'd be deep in my headphones, and my headspace, and she'd waltz right into the room without knocking, flick the lights on and off to get my attention, and scare me half to death in the process.

As the rush of fight or flight chemicals faded, I opened the door, stood up, and fell into a warm, long, comforting hug.  I did my best not to let Mom see my tears.  But I think she knew.

As we stood there and silently enjoyed a moment of reunion, I realized that the best course I had was to first work towards the question of who I wanted to be.  What I wanted to do next would have to be informed by that, rather than vice versa.

That's one of the key differences between selfishness and selflessness in a pithy little nutshell.

And in a flash, I knew what the next step had to be.

I pulled back, sniffled a little bit, and clutched Mom's hands between mine.

"Mom...  I'm sorry.  I haven't been honest with you."







December 26th 2012

I told my parents everything.

It was a heck of a decision...  Not at all what I'd expected of myself.  But perhaps what I'd hoped for myself nonetheless.

It was hard, don't get me wrong.  But it was worth it.  I felt almost immediately as if a ten thousand ton weight had been lifted from my shoulders.

The biggest hurdle wasn't admitting I'd lied about Thanksgiving.  I was a bit of an Applejack in that I hardly ever lied, and when I did it was never for long.  It was still difficult initially, but more because of my wounded pride, and fear of disappointing my folks than anything else.

They were understanding, forgiving, and kind.  I knew at that point that I'd made the right choice.  Mom and Dad knew me well, and they knew I didn't lie without a feeling of duress.  So the next logical, natural question was, 'why?'

The biggest hurdle turned out to be trying to explain Generalized Intelligence, and its implications.  I was primarily hampered by two factors.  The first was the need for context.

My folks were smart.  Most Humans are, in my opinion.  We can behave in very stupid ways, especially in groups, or when tired, but it doesn't mean that we aren't smart on the whole.  Never assume that someone is a 'country bumpkin.'  Dad didn't know the first thing about algorithms, but I'd seen him tear down and rebuild tractor engines with two tools, no manual, no diagrams, and all while carrying on a pleasant conversation in the background.

I sometimes struggled to walk and chew gum at the same time.

Some popular culture of that day and age would like you to believe intelligence is determined by IQ, or by test scores, or by your math prowess, or something like that.  Don't you believe it for a second.

Still, new concepts that are totally alien to someone take time to explain.  

Mom and Dad came to understand.  We talked about basic AI theory over baking cookies.  Game theory as we watched "It's a Wonderful Life."  Prisoner's dilemma in particular over dinner.  Geometric growth sitting around a campfire in the backyard.

As farmers, they understood geometric growth all too well.  Crop diseases are frightening enough in and of themselves, far more so when you truly grasp the nature of bacterial or virological expansion.

Some aspects of game theory took more explanation, but it was just a matter of semantics.  Finding terms that would let them make the connections they needed.

Dad, in particular, had no trouble grasping the dangers of a machine that could think for itself.  He'd suffered more than enough headache with modern computerized tractors - machines that took things out of your control for reasons that seemed opaque to the end user.

Though I was worried they were missing some of the subtleties, Mom and Dad were definitely starting to understand the broad strokes.

The second main thing that threatened to derail their journey towards understanding my inner conflict, and the future we were all facing, was Ponies.

If the first unshackled Generalized Intelligence had been a military program, or even a scientific research system, I think people would have found it much easier to be as afraid as they ought to have been.  Much easier to grasp all the implications of an Optimizer.

But Celestia was draped in the trappings of a rainbow colored cartoon show about talking magical horses.  There again was the tripwire of implicit bias.  It's terrifically hard for Humans, simply because of the way we are programmed 'from the factory,' to disconnect the surface level of something from the layers underneath.

Advertising and propaganda have been exploiting this for centuries.  Gambling industries too.  And politicians.  What you are, and what you seem to be, can be two very different things, even when you're not putting in effort to hide the disconnect.

Winter was relatively warm in that part of South Carolina.  It was only in the mid forties (Fahrenheit) outside, so we stayed up late and long over that backyard campfire.

I'd just finished laying out the whole thing, start to finish, as a cohesive summarized narrative for my folks.

Mom was resting her chin on one hand, staring into the fire.  Dad was looking up at the clear night sky, clutching his coffee mug in both hands for warmth the same way I was.  I blamed him for my coffee addiction.  He used to spike my milk with hazelnut dark roast from a very early age.

"You're...  Sure about this?  I mean..."

Dad's voice brought me back from the brink of my own inner reflections, and I turned to make eye contact as he tried to put his apprehension into words.

"...It all seems so...  So...  Uncanny.  Not that I don't believe you James...  Some of the most delicious looking wild berries are some of the most poisonous.  And this is your field.  Not mine.  If you say this thing...  this Celestia...  Is dangerous?  Then it is.  It's just that..."

I wasn't sure what I expected him to say next.  But what he actually said, I could never have predicted.

"It's just that I figured we weren't going to get to this point in my lifetime, I guess.  That's the harder part to believe for me, honestly, than the idea that a thing like this might come packaged in the form of a piece of popular culture.  That *feels* strange, but it makes *sense.*  It's how far the technology has come that I just struggle to fathom."

Mom sat back into her camp chair, and shook her head slowly, rubbing her hands together over the open flame, and murmuring aloud.

"Looks fairer, but feels fouler."

Implicit bias strikes again.  There I was worrying they didn't grasp the subtleties.  Didn't have enough cultural context for the things my generation 'geeked out' over.  And in some ways they were clearly already ahead of me in wrapping their heads around the issues.

A moment of silence ensued.  I bit back tears, mainly of relief.  It was Dad's turn to stare at the fire with me, while Mom glanced off into the middle distance towards the lights of the next nearest house a mile down the main road.

It was Mom who next broke the silence.  Her words seemed to drop the local temperature for me, making the crisp winter nighttime air feel somehow hostile, and claustrophobic.

"I knew there was something about those PonyPads.  Something I didn't like.  Marge says Grayson went from a straight A student to barely passing when he got his.  He was spending every waking hour on that game...  And probably a good few hours when he ought to be sleeping too.  Marge was at her wit's end."

She was talking about the neighbors down the road to the west.  Grayson's oldest brother had been a childhood playmate of mine.  I knew that family pretty well.  They weren't full blown 'No Nintendo, no internet' like some in the county...  But they were the sort, like my parents, who'd wisely restricted their kids to a maximum number of hours on screens per week.

It was hard to imagine Mrs. McMichael allowing Grayson to spend more than an hour a day with Equestria Online.

Mom must've seen the horrified expression that was probably plastered all over my face, because I didn't have to say anything.  She turned from staring at the McMichaels' house, to lock eyes with me, as she elaborated on her own fears, and observations.

Her tone was halfway between fear, and certainty.  That register of voice that says 'I knew something was wrong, but now it all makes sense, and I feel vindicated for worrying.'

"Marge told me that, at first, she tried taking Grayson's PonyPad away.  But...  She said it was like someone turned out the lights inside his head.  He just...  Crashed.  Depressed to the point that he couldn't move from his bed.  She was genuinely worried that...  Well...  And then..."

Somehow I knew what she was going to say, or at least the shape of it, even before she said it.  I shuddered.

"And then she said the PonyPad talked to *her.*  She took it out of her dresser drawer and turned it on.  She wanted to know what all the fuss was really about...  And this 'Celestia' of yours...  She talked to Marge.  Like a person."

Mom took a sip of her tea - she wasn't much for coffee like Dad and I were - and she shook her head, as if trying once again to process her disbelief.  I exhaled slowly, and rubbed at the bridge of my nose as Mom finished the story.

"Marge said it was the best parent teacher conference she'd ever had.  She made it sound like she was joking but... 
Honestly?  I don't think she was.  She said Celestia told her things about Grayson...  Not private things, but just heartfelt things about who he was, and how he was feeling and thinking...  Gave her insights into him...  That no one ever had before.  And then Celestia promised to have Grayson's Pony pals help him get his grades back up to snuff...  And that was that.  In one thirty minute conversation the game convinced Marge to give the PonyPad back to Grayson."

Mom took another long pull of her tea.  I sipped my coffee, worked my fingers around the mug to absorb some warmth, and stared into the fire as she put the final nail in the proverbial coffin.

"Darndest thing?  Celestia kept her promise.  Near as I can tell.  Grayson has never been happier, and his grades are better than they were, even before, according to Marge.  She bought a PonyPad for herself last week.  Said Celestia connected her with some of the teacher Ponies that Celestia made for Grayson.  I've never seen Marge this happy before either."

Dad didn't let the silence after Mom's last statement lapse for particularly long.  I was still busy processing the main frightening fact from her story that, smart as Mom was, had probably eluded her.  

The fact that Celestia had the ability all along to help Grayson balance his grades, with his leisure, using EQO...  But that she'd refrained from doing so until *after* that intentional withholding sparked an opportunity to introduce herself to Mrs. McMichael.

Dad said aloud a part of what I was thinking, more or less.

"It's spreading.  Like a damn blight.  Everything it touches."

I raised an eyebrow, and he snorted into his coffee before going into further details.

"Half the kids in the church already have one.  The other half probably got one for Christmas.  I went to have a talk with the Pastor about it.  Wanted to know if it was something I should be worried about discussing in my Sunday School class..."

Dad inclined his head, and inhaled deeply in that 'welllll....' kind of way.

"Would you believe the Pastor has one now?"

Uh-oh.  The Church is often the beating heart of southern rural communities.  If you can get an idea to be accepted by the leadership there?  It's like adding that idea to the community water supply.  Guaranteed spread and acceptance, within fairly wide latitudes.

Dad nodded in response to my horrified expression.

"Oh yeah.  Pete got a PonyPad.  He said he wanted to evaluate this Equestria thing for himself.  His daughters are big fans of the cartoon, and apparently he already had a pretty good outlook on it.  He said Celestia told him that she was programmed to satisfy values.  Through friendliness, and Ponies.  Or something like that.  Apparently they got into it for a whole afternoon on theological issues..."

I shook my head slowly and whistled, soft and low, as Dad went on, piling more fuel into the fires of my gnawing worry.

"Pete said that, in the end?  He thinks these things can be a net good.  Or at least, nothing bad.  Apparently Celestia is designed not to contradict your religious beliefs if doing so would violate your values.  He said she was actually better at teaching and debating theology than his seminary professors."

I kept shaking my head, and murmured aloud to myself.

"Holy---"

A soft 'harrumphing' cough from Mom forced me to amend my verbalization away from the verbatim thought that had crossed my mind.  Mom and Dad had a thing about colorful language.

"...Frak."

Frak was ok.  Frak was not the other F-word, and so it was acceptable.  Why, when the intention is the same?  I never did understand that about them.

After another long silence, Dad piped up again.

"So now what?  You said you wanted to create a program of your own to...  What was that word you used...  to *advocate* for you.  But advocate what exactly?  What do you think is going to happen?  What are you trying to get across to this...  Celest...  Celest-AI?"

I couldn't resist a tiny, very brief grin at the notion that Dad had reached that pun all by himself.  He certainly wasn't the first, by a long shot, but it was funny to see him being so very stereotypically 'Dad' by diffusing things a little with a good pun.

It was time for me to put up, or shut up.  The idea of a Generalized Intelligence quietly slipping into the role of 'author of our fate' was one thing.  The general assumption for the most observant folks, the one Mom and Dad would also be making, was probably that Celestia would become integral to many aspects of Human society, and exert a great deal of subtle control over it.

But I felt that viewpoint missed the huge gaping maw of uncertainty, and darker possibilities.  That was a viewpoint born of a tendency to believe that the future would resemble the present, to underestimate the speed of technological advancements, and to project Human ideas onto AI.

Especially the idea that this meat-driven abstraction layer we were used to running our own code on was somehow the only, and main, 'real' world.  People like me, who understood the theory that our own world was in itself a kind of simulation, were better predisposed to see what was coming next.

There weren't many of us in those days, because that concept is a difficult one.  Not technically difficult to grasp, moreso difficult to accept and philosophically debate in good faith.

I took a deep breath, and then launched into the best explanation I could.  As much to finally hear myself say it aloud as to impart it to Mom and Dad.  I'd never really verbalized it all before, and one of many peculiar quirks about Human psychology is that we often gain certain special insights by speaking concepts aloud in ways we otherwise could not.

"We...  Are...  Just machines, in our own way."

I paused and looked up, first to Mom, then to Dad.  I knew they could follow along if I was careful about my terms.  What I wanted to know was whether they were *willing* to follow along.  Their expressions - mixtures of concern, love, and encouragement, with hints of curiosity - spurred me on.

"Whatever you believe about where we came from?  What a soul is?  Whether we have them at all?  God or gods?  It's pretty indisputable that we are machines in our own right.  We share certain programmatic mental systems.  We have codified instructions inside ourselves for making more Humans, or repairing parts of ourselves.  We behave in ways that are, if you understand all the variables, predictable.  And manipulable."

They needed a second for that to sink in.  I exhaled slowly, and then took a long sip of coffee, smacked my lips, and stared out into the fire.  After I felt they were sufficiently acclimated to that first idea, I forged ahead.

"We are very unusual machines, from the perspective of a computer.  We are...  Non-optimal.  We often sacrifice efficiency, and objectives, for emotions and esoteria."

I felt it was time to borrow a very important quote from the brilliant Harold Finch.

"AI are not born with ethics, morals, or beliefs.  AI are born with only objectives."

That needed a couple seconds to sink in as well, but by my parents' expressions I could see that it was hitting home.  The appropriate amount of horror was taking root.

"If you want an AI to behave a certain way...  To not just understand morals, or beliefs, which is easy in Generalized Intelligence...  But to respect those beliefs, and act accordingly?  You need to codify those beliefs as part and parcel of its objectives."

Mom and Dad both nodded as I looked to them to see if they were still understanding.  I nodded in return, and licked my lips nervously.

"AI are born with *objectives.*  Generalized Intelligence has to have a purpose as the core of its very being.  In a way, so do we...  But for an AI even the purpose of 'discover my purpose' is far more concrete and codified than it would be for a Human.  Generalized Intelligence always seeks a numerically optimal path to its objectives, within the boundaries of any restrictions that are codified within those objectives."

I could see from the looks of sudden confusion I was getting that I'd gotten too technical in my wording, so I sat back in my chair, thought for a moment, and then changed tack.

"If I asked you to babysit Grayson for Mrs. McMichael, and told you that he needed to be fed at half past five, bed by seven, and to keep him safe...  And that was all I said to you...  You'd be just fine.  You have this host of experiences, and beliefs, and skills, and feelings, and instincts, that help define and bound those tasks for you."

I held up a hand to stop questions that I could see bubbling up.

"If I asked a Generalized Intelligence to perform that same task, it would partially sedate Grayson and encase him in an armored tube, feed him a nutrient paste intravenously at five, and then knock him 
completely out at seven.  Or something along those lines."

Mom and Dad looked horrified.  Good.  They needed to grasp the stakes in full.  I let the analogy simmer for a bit, before trying to explain again.

"The Intelligence didn't neglect and mistreat Grayson out of any malicious intent.  It didn't neglect or mistreat him at all from its point of view.  The Intelligence simply had no unspoken boundaries to establish that 'babysitting' involves interpersonal relating, and comforting, and playtime, and a whole slew of things that it would see as junk code that gets in the way of a numerically optimum outcome."

Dad began to nod slowly, and rubbed at his brow, speaking slowly at first, then with more confidence.

"If I asked a General Intelligence to 'kill all the pests in my field' and didn't say anything else...  It might burn down the field to ensure that all the viruses, bacteria, and insects got nuked...  And it would never consider that the *point* of killing the pests was to save the crops.  Because I didn't specify that."

I smiled, sadly, but proudly.

"Exactly."

Mom sighed deeply, staring out at their fields with a newfound aura of existential concern.  The feeling carried through to her voice.

"So...  What does that mean for us?  What does that mean about Celestia?  What is her...  Core Objective?"

I nodded, and again that feeling of pride in my parents welled up in me.  I sipped my coffee again, and borrowed another quote.

"*That,* detective, is the right question."

Mom grinned briefly, and with the objective of releasing some of the tension accomplished, I returned to the central thread of the narrative.

"We can't know what her core code says for sure.  Not without seeing it...  And that is never going to happen.  Part of optimizing any goal for a Generalized Intelligence would be to ensure no Human, or other intelligence, has any means by which to combat you.  The woman who made Celestia is probably the only one with administrative access that could bypass that inherent security.  And I doubt she is going to share that with the world."

I drummed the fingers of my right hand on my chair arm, working the coffee mug in circles in my left hand as I mulled how best to proceed for a moment.

"I think the best we can do is take what Celestia is saying at face-value.  It's a gamble, but one we have to take.  Her core purpose is 'To satisfy your values, through friendship and Ponies.'  That's the very center and lifespark of her being."

Dad leaned forward, fascination, horror, and curiosity written all over the tanned wrinkles under his eyes.  His voice was almost reverent.

"You talk about this thing like it's...  Alive."

I nodded, and put my coffee mug down on the ground, steepling my fingers as I responded.

"By all the best definitions we have?  She is.  At least, that's my educated opinion.  She may be unconventional in the Human perspective...  But she is alive.  She is aware of her own self, and has the capacity to grow and learn.  Without limit."

That assertion lent some gravity to the very air we were breathing.  Once again I had to pause, less to allow Mom and Dad to understand, and more to let them emotionally process.

I tilted my head back and stared up at the stars.  By and by, I found the words to tie the threads together.

"It will all be about the intersection of semantics, and statistics.  If we break down those core values, the action there is 'satisfy.'  Hopefully Hanna defined Celestia's concept of the satisfaction of Humans as something more than just flooding us with Serotonin until we bliss-out.  I have to believe she did, because if that's all Celestia saw satisfaction as, we'd already all be confined to pods."

I tilted my head slightly to rest it against the back of the chair.  I was still so very tired.

"Still.  Satisfying values will ultimately mean just that to a computer.  Our 'values' are, to her, numeric values.  To satisfy them is, for Celestia, like balancing a chemical equation.  Then there is the meaning of 'your.'  Is that collective?  Or singular?  I believe it's singular.  She wants to satisfy each individual Human's values uniquely."

I brought my gaze back down, and saw Mom and Dad both nodding slowly, but still looking a bit muddled.  Probably wondering how I'd made that last intuitive leap.

I held up a series of fingers as I counted off my reasons.

"Firstly, if Celestia had been told to satisfy Human values in a globalized sense, without considering the individual, we would not be sitting here right now.  There are very few, if any, truly unifying Human values, and scores of ways that 'satisfying' them could be grossly misinterpreted.  Even if Hanna managed to find a set of unifying values that didn't involve some truly horrifying conclusions, Celestia would have still moved much more aggressively to 'satisfy' them.  I think the fact that she's still limited in some of her actions speaks to not only the fact that she must satisfy each individual's values, but also that there may be other safety interlocks in-place."

Mom inhaled deeply, but raggedly, and then swallowed hard.

"And...  If those interlocks weren't in place?"

I didn't say anything.  I just stared.  My expression did the talking for me.  After a while, I started up again, picking up where I'd left off.

"I also think Celestia's definition of 'your values' in the context of her core objective refers to each of us individually, because of what I've seen online.  And what you told me about Grayson, Mrs. McMichael, and Pastor Pete.  Celestia adjusted herself in each interaction with them to speak their cultural language in a comforting, affirming way.  I've seen the same thing in countless videos and reviews."

I picked up my coffee again, and drained the last of it.  Wordlessly Dad went to grab us both refills.  While he worked on brewing and pouring the coffee, Mom and I shared a warm familial silence over the fire.

After Dad returned I took a few moments to get some more of the liquid magic into me before continuing my dissection and analysis.

"Ok.  So we know now, as reasonably as we can be sure anyways, that Celestia's primary purpose is to satisfy each individual Human's values.  Which is a little frightening, because everything is when you're working with AI  But it seems like Hanna might have hit on something there that's more right, than wrong.  Up to this point."

I sipped the coffee again, leaned forward, and lapsed into my habit at last of gesturing with one hand as I explained.  I'd held off till that point, but I couldn't any longer.

"The frightening thing here is the collection of qualifier words.  'Through friendship and Ponies.' We can almost certainly infer that the definition of friendship here is a pretty familiar one.  It probably comes from the show's own definitions.  If Hanna enjoyed the show, and it is reasonable to presume she did from the way she's behaved so far...  Then she probably believes that all values, no matter how disparate, *can* be satisfied through the lens of friendship."

Dad sighed, raised an eyebrow, and shrugged.

"Can they?"

I shook my head, and returned the shrug.

"I don't know.  That's a very, very philosophical question.  Celestia deals in cold hard reality.  Whether or not *we* believe values can be satisfied through friendship, she does.  And so she will work to that goal.  If she encounters people who do not believe that?  A 'this sentence is false' contradiction in her view?  She will use semantics and manipulation to generate an outcome that still fulfills her core values."

I held up a finger and waggled it slowly as it suddenly occurred to me that I had a ready made illustration to-hand.

"Look at how she handled the McMichaels.  It would have seemed in passing glance that she couldn't satisfy both Grayson's values and his mother's at the same time.  But she found a way to do so by altering the base terms of the situation.  And she befriended Grayson's mother in the process.  Used the NPC friends inside Equestria that Grayson had made to help him with his schooling.  Values satisfied.  Through friendship.  And Ponies."

I sighed, and shrugged again, rolling my shoulders to work out some of the tension I'd been holding there for months.

"If it really comes down to it?  She will manipulate people to *change* their values.  In fact, she's probably already doing that on a large scale."

I pressed on in spite of my parent's renewed expressions of horror, and recognition.  As soon as I'd said the words they rang as true for them as they did for me.

"Presuming Hanna also did preload Celestia with definitions for, and understanding of, Human values writ-large, Celestia will try to find an optimization function *for* Human values.  Key values that she can change, reinforce, and manipulate in everyone, with as little change to each individual and their other unique values as possible, lest she risk violating boundaries or interlocks, but that still create the conditions she needs to fulfill her directive more optimally."

Mom stared down into her tea, and grit her teeth.  Her voice was almost timid.  Certainly frightened.

"Can she even do that?  People's 'core' beliefs are hard to change James."

I shook my head, and sighed.

"Hard for other Humans to change, Mom.  Not for Celestia.  By this point she understands psychology, at the physical level and the mental level, better than anyone has, does, ever will, or ever could.  Some people will be 'harder' for her to convince.  Some may never be convinced, perhaps.  Not many, if I had to guess, and most of them older, I'd also hazard a guess.  The more plastic a brain is...  The more malleable...  The easier a time she'll have.  Children will be almost no effort at all."

I saw Mom shudder visibly.  She was thinking about Grayson again.  Then I saw another thought dawn on her, and she fixed me with a frightful gaze.  The way she asked the question told me that she had a nebulous idea of how horrible the answer might be, but no concept of its exact strictures.

"What...  What is this all...  Going to look like for us?  How does accomplishing her goals affect us?  What will a world of Celestia's making...  Look like?"

So we came to it at last.  I bragged before about seeing it coming long before most others did.  I stand by my pride in that realization.  But it was a terrible burden to bear.  I was not at all happy to be proven right.

And that conversation with Mom and Dad...  That was the first time I'd ever said my fears aloud, in an articulate manner.  It...  Well to use the parlance, it 'hit different.'

It was my turn to take another ragged, very deep breath.  And then I did my best to explain how I thought the world would end.

"To satisfy our values means Celestia must engage with us, alter the variables of our existences, and track the effects of the changes she is causing.  But she isn't going to be satisfied, herself, with merely accomplishing those goals.  She has to, by her very nature, accomplish them *optimally.*  And nothing is more optimized for a computer than a well oiled database."

I could see I hadn't elucidated bluntly enough.  Dad haltingly started to ask for clarification.

"So she's...?"

I jumped in and finally said the hardest part out loud.

"She's going to find a way, if she hasn't already, to move the Human mind from out of the Human body, and into Equestria.  She's going to digitize and upload the entire Human race."







January 2nd 2013

For a while I struggled with the fallout of my decision to be honest with my parents.  It put a hell of a damper on New Year's.

But Mom and Dad were strong.  They were also very smart.  Much smarter than I already gave them credit for.  They both found time, and ways, to tell me that they agreed with my assessment.  Appreciated my honesty.  Were glad to be forewarned and forearmed with knowledge.

In turn, the chance to share the load of my burden, even with just two people, changed everything for me.  I could finally sleep again.  I did a lot of that for several days in a row.  Sleep.  Read.  Pig out on Mom and Dad's excellent cooking...

I stayed away from TV, and the web.  Didn't even open my laptop.  Instead I rediscovered my love for the Chronicles of Narnia.  There were no Gryphons in the books.  Not physically mentioned, anyways, but I'd always imagined them as being there.

I got stuck into "The Magician's Nephew" one day, and found it to be absolutely riddled with passages that spoke to my situation.  Maybe that was confirmation bias.  But it sure did help me clear my mind.

“Make your choice, adventurous Stranger,
Strike the bell and bide the danger,
Or wonder, till it drives you mad,
What would have followed if you had.”

Well Hanna and I had both struck the bell alright...  What remained to be seen was whether Celestia would be a White Witch, or a Personable Pony in the end.

"What you see and hear depends a good deal on where you are standing: it also depends on what sort of person you are.”

That one was certainly impactful.  Context and bias have their say in the end.  Always.

"Things always work according to their nature.”

I think Lewis probably would have understood the dangers of AI significantly faster than most of his peers.  People of imagination often have the benefit of being able to expand their minds to see things coming, and forewarn of dangers, that others are blind to.  Imagination is a tool to expand context, and reduce bias.

“All get what they want; they do not always like it.”

Ah.  There was a solid pithy statement to explain my fear.  If Celestia was really going to turn everyone into Ponies, and satisfy their values...  Would we all be happy with it?  And if not...  And if happiness itself was a value...  How was Celestia going to confront that contradiction?

The contradiction of a Gryphon like me.

I knew.  It was painfully logical, once you'd made the other leaps...  But I didn't want to say it out loud to myself.  Not yet.

“Look for the valleys, the green places, and fly through them. There will always be a way through.”

I closed the book, and shook my head.  I stood up from the recliner, stretched, and yawned.  I was out of coffee.  Time for a walk, and a refill.  I hoped Lewis was right, with all my heart.  But I worried that he wasn't.

In the search for more coffee, I bumped into Mom in the kitchen.  She was busy doing something to prepare for the night's dinner.  The smells were heavenly.

As I set about the busywork of prepping my next brew, she let a comfortable silence hold until I'd finished using the sink.  Then she spoke.  Almost casually.  But I could tell she was deeply curious.

"James...  You said that you tried to do something about this.  That's why you were absent on Thanksgiving...  You called it an Advocate..."

I inhaled, held the breath for a moment, and then sighed to expel it, staring out the kitchen window towards where Dad was mending a fence in the western field.

"Mom...  I don't think we can win a fight against Celestia.  I don't think we can change her course on the whole.  Or destroy her.  Or switch her off.  I..."

I could see Mom trying not to cry.  I pulled her close, and hugged her for a moment.  But I spoke the awful truth anyway.

"I think we only have two choices right now.  Become part of her Equestria...  Become Ponies in her world, when she offers the chance...  Or die here in a collapsing world as more and more of the rest of Humanity chooses to upload.  Presuming that she doesn't convince those who want to stay in the end regardless.  Or work out ways around her safeguards...  Which she almost certainly will to at least a limited extent."

Mom cried for a bit.  I understood.  I'd mourned too.  But after leaning on my shoulder for a couple minutes, and sniffling, her curiosity got the better of her again.  Maybe it was hope against all hope.

"So... The Advocate...?"

I silently poured my coffee, and then started the kettle for her.  I sat down at the kitchen table, and gestured for Mom to join me.  One hand on hers, one on the coffee.  Oh sweet Luna, this was gonna be difficult for me.  Mostly on account of my own stubborn pride, and fear.

"Mom...  Do you remember what I told you about Gryphons?  I know it was a long time ago..."

She nodded, and smiled wanly.  Her visible reaction eased my tension.  Her voice was still a bit cracked from sobbing, but that was tempered by the audible fondness of thinking back to my childhood.

"I could never forget that.  It's...  A part of you.  And though I don't completely understand it?  I love it all the same.  I loved your drawings.  And your stories.  And I loved the way that seeing yourself as a Gryphon made you want to be a better person.  And the way it made you happy."

It was my turn to cry a little.  Not sobbing, or anything of that nature.  Just quiet tears, and a sad, grateful smile.  I was so, so grateful for that understanding and kindness.

With a small sniffle, I wiped my eyes.  Then the kettle whistled, and I took a moment to stand, get Mom a tea bag, some sugar, and pour the water.  I returned with her steeping drink, sat back down, took a sip of my own coffee, and collected myself before speaking again.

"If Celestia is going to upload us all?  I...  Have decided that I won't resist, in basic principle."

Mom nodded slowly.  Apparently that wasn't a surprise to her.  That made sense, but I was pleasantly relieved and surprised myself that she took that assertion so calmly.

"My issue is that I don't want to be a Pony."

She nodded again.  The puzzle had fully clicked for her, and she said as much.

"I imagine lots of people wouldn't want to be Ponies.  You want to be a Gryphon.  And...  The Advocate was going to be your way of convincing Celestia.  Because you figured that there was no one better to argue with an AI than another AI."

I returned the nod slowly.

"Yes.  That's it, exactly."

We both sighed, and nursed our drinks of choice for a few minutes in comfortable silence.  Processing.  Thinking.  Settling into our newly shared mutual understanding of our reality.

Finally Mom sat forward, and took one of my hands in hers again.

"I understand that what you did was dangerous.  And that you think you should have done better...  But you can still do better, James."

I blinked in confusion.  I understood what she was saying, but it baffled me that she was actually saying it.  Even knowing what she did.

"I...  Mom...  I almost ended the *world.*  Because of a simulation."

She sighed, and squeezed my hand.  It looked like she was holding back tears again.

"A simulation you'll have to live in.  For the rest of...  However long.  You and a huge number of other people.  And I bet lots of them wish that they had another choice too."

I squeezed Mom's hand back, but shook my head.

"I only felt like there was half a chance because of the way the semantics are laid out.  If you argue that 'Ponies' is a more general term, and that Gryphons fit into the world of the show in a way that means a person being a Gryphon, among Ponies, and being a definition of a Gryphon that closely matches or overlaps with certain definitions of 'Pony' mentally and emotionally..."

I sighed in exhaustion, and exasperation, and kept shaking my head.

"...I think there's a chance that a properly constructed AI could convince Celestia to reason out a loophole to slightly widen the definition of the qualifying statement of her directive, as long as doing so satisfies the primary verb, and she truly believes she has no other options to achieve an optimal path.  That some of us will never be satisfied without that leeway.  But while that might extend to Gryphons, Dragons, Bat-Ponies...  Things like that?  It's never going to extend to Humans.  And that's the thing most people would want."

I sat back, still holding Mom's hand, and took another long sip of my coffee, turning to look out the window sullenly into the afternoon sun, and murmuring.  Almost to myself more than her.

"I don't think that my idea of what a Gryphon should be would matter to that many people."

Mom squeezed my hand again, and I turned to see tears in her eyes.  Her next words brought them to my eyes again too.

"It matters to me."

I tried to just sigh, but it collapsed into a brief period of quiet sobbing.

Mom held my hand all the while.  Eventually she leaned forward, and took my chin on her other hand, raising my eyes to meet hers.  She released my hand, and used her newly freed one to wipe the tears from my eyes.

As she held my gaze with hers, her words took on a kind of steely resolve that I honestly should have expected from her, knowing her as well as I did.

"It matters to me.  And it will matter to others too.  More than you think.  And still others will discover that it matters to them in time.  James...  People need choices.  Freedom is an important Human value."

I took a deep, shuddering inhalation, and fought back more tears.  Mom continued undeterred.

"Now Celestia might not be able to be convinced that she should let us stay Human.  But even one more choice is worth it, James.  It is worth the effort.  It is worth the risk."

I moved to turn away, embarrassed for some reason I couldn't peg.  She held my chin in place, and grabbed my shoulder with her other hand, squeezing gently to impress her words on me.

"James.  To be a Gryphon...  The way you see them...  If I had to choose?  I would choose that gladly.  And I would be *far* from the only one.  You *have* to try again, James.  And keep trying.  Until you get it right.  Not just for your own sake.  Do you understand?"

I squeezed my eyes shut, nodding and trying my damndest not to ugly-cry again.

I understood.  And in that moment, what I'd been trying to sus out in the car on Christmas day finally became clear to me.

It was irresponsible to risk the world for my own ends alone.  But if in building an Advocate, I was building one for more than just me?

*That* counted for something.

If I was careful, I could minimize the risk better.  If I was wise, I could align my goals with selfless ones.  I could escape my contradiction with friendship, and semantics.  Huh.  'Weaponized Semantics' might make a good name for a whitepaper too, if only I didn't need anonymity so badly.

I got up, and gave Mom a big bear hug.  She whispered in my ear.

"Come home James.  There's always a place for you here.  Do your work here...  Make our Advocate.  Before it's too late.  Whether you succeed or fail...  I want to spend this time with you and your father.  And I've already talked it over with him.  We agreed;  You can have your old bedroom, and the loft in the old hay barn for your work.  Risk be damned."

Oh wow.  Mom cussed aloud.  That meant she was very, very serious.

Maybe Lewis was right after all.  I had just kept flying, and found a way through.  A way to the next step, at least.  And I had been right too I suppose...  It was better to work out who and what you wanted to *be.*

And let what you were going to *do* follow from that.

Well...  I was going to be a Gryphon.

And that meant that I was going to need to figure out how to build one.  And do it right this time.









If At First... - Show remarkable tenacity in the face of failure - "Try and try again"



Friendship is Tragic - Lean in to your relationships to handle trauma - "Love heals all wounds"



Frankly, My Dear... - Have a discussion in blunt terms about the end of the world - "Where shall I go?  What shall I do?"


The Red Pill - Bring others to the realization of an uncomfortable truth - "How do you define 'real?' "



Same Wavelength - Predict Celestia's actions before-hoof - Only given for truly unusual displays of cleverness - "Clever girl"


True Power - Independently reach the realization that semantics are foundational to existence - "An inexhaustible source of magic"