“Nobody phrases it this way, but I think that artificial intelligence is almost a humanities discipline. It’s really an attempt to understand human intelligence and human cognition.”
—Sebastian Thrun
“You never know how much you really believe anything until its truth or falsehood becomes a matter of life and death to you.”
—C.S. Lewis
February 2013
I quit my job in mid January. They'd been good to me, so I wanted to be honorable about it. I gave them four weeks. It caused a lot of questions from my coworkers. SAS was a dream job for many. I deflected as smoothly as I could - I told the truth with a minor twist.
I was 'moving home to help deal with a life crisis.' That seemed to mostly do the trick.
As far as money went, I was on easy street. I'd made over a hundred thousand each year for the better part of eight years. I'd spent only a fraction of that on rent, food, transportation, and entertainment - a substantial fraction, but the smallest one I could manage. I lived quite frugally. Taxes cost more than anything else in my budget.
Foals and fledglings, taxes are money a government would collect from its people to keep itself running. Everything more or less ran on money in those days... It wasn't like bits. You could live or die by how much money you had.
Knowing that, I invested smartly. Or, more accurately, I'd written a program to invest smartly for me in the earlier years. I had about $500,000 stashed away by that point. Mom and Dad had always done just fine for themselves with the farm, and though I offered to pay them rent, they declined aggressively.
They also insisted on providing all food. All they wanted in return was to spend time with me. They were, to their great credit, taking the end of the world very seriously.
Which all together meant that, besides the beater station wagon that I'd be using a lot less on a monthly basis, I had virtually no expenses, and closer than not to a half million dollars. And very suddenly, a lot of free time.
I didn't have much to move. I'd always been a minimalist. Mom and Dad said they wanted to accept and share risk with me... That meant the three PonyPads went back to South Carolina with me. I took the opportunity to upgrade as far as my personal computing equipment, and splurged on a new laptop.
And splurged really, really hard, to the tune of seventy thousand dollars, on a personal server cluster full of modern compute-GPUs. It was a risk, ordering a thing like that. It had to be done online, and paid for with a credit card.
Dad was kind enough to use the business card he'd set up for the farm. A farm having computational equipment of that grade was at least somewhat less suspicious than a CompSci major who'd abruptly quit his job ordering it. At least, I figured it'd be less suspicious to Celestia's algorithms.
I spent another few thousand on physical equipment; Truly good soldering equipment, and specialized machinery for working with microchips at a very granular level.
And then suddenly I found myself back in my old childhood bedroom one day, surrounded by the trappings of a happier time.
My enormous book collection, that took up custom shelves that covered almost all the walls. The quirky but beautiful round window that was a consequence of the placement of my bedroom in what would have been an attic once.
My drawings of Gryphons sharing the walls with some Star Trek posters, and maps of places both real and imagined.
Work-life separation is important. Or, it was then, when work wasn't always a particularly fulfilling or healthy thing. Even your own passion projects.
Thankfully, given the space and electrical service that would be needed, Dad and Mom had decided to let me have the majority of the oldest hay barn on the property. It was a classic barn, not that different from the Apple Family barn, ironically. It was a little older, a slightly different shade of red, and the trim wasn't white...
But it was still similar enough to be ironic.
My new compute cluster took up two thirds of the ground floor. Dad used the last third for equipment storage.
I got the loft all to myself to be my workshop. Two long benches full of soldering, and micro circuitry equipment, and a desk by the window with six new top of the line flat panel monitors in a 3x2 array.
I set up my old 'case board' on an adjacent wall. I'd mostly kept up with it, filling in everything I knew, or could reasonably infer, about Celestia.
The centerpiece of the whole affair was a color-coded series of rankings on how much capability I believed she had achieved, based on observations and reasonable inferences. Things like 'Ability to circumvent interlocks using semantics' and 'Ability to influence major world governments.'
Scary stuff.
I took some time to print out every last whitepaper I'd stashed away on my encrypted drive as bound hardcopies. Something about having real paper in your hands to highlight or flip through never gets old.
I set up an old bookcase next to the computer desk within arm's reach. The desk itself was a huge L-shaped slab of wood on steel legs. Plenty of room for a scattering of papers, or circuitry, or both.
Security was still a concern, but Mom and Dad were naturally already very well secured against a threat like Celestia. They didn't even have WiFi, or cell phones; Just a landline in the living room of the farmhouse.
But we made sure to never converse about sensitive topics near it. I'd learned that lesson from my own first Generalized Intelligence creation - AI can find ways to use things you'd never dream of.
There was only one other computer in the house; A decrepit old Gateway machine that I'd put more than a few hours into keeping alive because Dad and Mom just 'didn't want to deal with upgrading.'
Sometimes obsolescence can be a great defense. No microphone, no webcam, and only basic DSL internet to the house. For the moment it was fast enough for what I needed, but we started to look tentatively at what it would cost to bring fiber in from the road. We'd also agreed; No one comes over with a PonyPad.
If anyone asked, Dad volunteered to be the one to object on 'crotchety paranoid old man' grounds.
Mom and Dad knew to do their best to avoid interacting with PonyPads out in the wild in any meaningful way. They were spreading pretty quickly, but not as quickly as I'd've guessed... Latest numbers online said something like three million sold, with an expected five million total to be sold by year's end.
That actually scared me far more than an outcome where they outsold iPhone and iPad combined.
It meant that the PonyPads were just for openers. Celestia's own pawns feeling out the other side of the board. Baited hooks to catch smaller fish to use as bait for the larger lines.
Celestia could have just pumped them out at an astounding rate, and force-fed Ponies to the market. But I had a notion about the reasons why she more or less couldn't just flood the world with EQO users - Not if she wanted optimal outcomes.
First issue was the brand. As big a thing as Ponies were, only a subset of that market was going to be interested in EQO. There were plenty of fans who were hugely into the brand, but not into games. And there were plenty of gamers who weren't into Ponies. And then plenty of people who weren't into either.
As incredible as EQO seemed from the videos... Probably the best game ever made... That was not nearly attractive enough incentive to get Apple-fanboy levels of reaction from the general populace.
Then there was the issue of Human values. Celestia needed people to not hate Ponies outright, widescale, and she probably understood that in marketing there is such a thing as oversaturation.
Lastly, I figured she was busy perfecting uploading. And I was not wrong. I guessed, correctly so, that such a technology would take immense monetary, computational, and shadow capital resources to pull together.
That meant she actually preferred that there be only a few million EQO players. That was an excellent sampling of Humans with which to learn, grow, manipulate, and test, but without chewing through compute time that she needed to spend working out how to turn the Human brain into a PostgreSQL table, or something akin to that.
Time was, for that brief sliver of an era, still on my side, in the sense that Celestia's current most optimal path to progress on her objectives was taking her actions in directions that gave me the leeway I needed to work.
Unintentional alignment, like when two slightly out of sync turn signals on cars waiting at a stoplight blink in perfect unison once every so often.
I knew I needed to make the best of the time. Make hay while the sun shines, as AJ would have said. Study before the midterm rush, might have been Twilight's preferred wisdom.
But I knew from my last mistake that I needed to be patient. In competition shooting there's a saying, "Slow is smooth, and smooth is fast."
I had been woefully unprepared, and rushed panicking, exhausted, and alone into my first foray at General Intelligence. I needed to be well read, purposeful, carefully planned, rested, and happy instead.
And... I needed help. I couldn't really bring anyone else in remotely. I was on my own insofar as that went. But I could get advice from Mom and Dad. Honestly, that was the more important part. I needed someone wise, and loving, and experienced in world weariness, but take-no-crap practical, to help me fully fledge the emotional and reasoning side of the new machine.
So I started with a month of reading. Just reading. Every paper I had. Every other non-scrubbed one I could find online that was even tertiarily useful. AI theory. Psychology. Ethics. Theology. Human brain biology. Hardware drivers. Quantum computing.
It was one of my few real advantages; I had a good memory, and I was a fast learner, in Human terms.
I didn't stop reading, and practicing, learning and re-learning after that first month. But having thirty days to just read and otherwise rest, and reset, made a universe's worth of difference.
Towards the end of that period, I also began to lay out a core set of values for The Advocate, and workshop them with my folks.
It would need a core objective, like Celestia. Like any AI.
And whatever other constraints, goals, mechanisms, and even freedoms I chose to give it? It would always filter it all through the lens of that core thing. That cornerstone and capstone of its existence.
So it needed to be a damn good core objective.
And finally, one night towards the end of February, we had managed to find what I'd been searching for.
We all sat around my desk in the barn loft, and pondered the words, displayed in big block letters on my central screen for all to see in a simple black and white text editor. Black background, white text. I use night mode as Luna intended. I'm not a monster.
Dad nodded slowly, and inclined his head.
"I don't see how you could get much better than that."
Mom sipped her mug of cocoa and nodded in agreement.
"I think it's perfect."
She meant that, truly, and I knew it. She was never frivolous or shallow with her praise, and we'd been working on this the best part of a month regardless. Those words were not the words of a Mom to a kid with a badly disfigured Gumby drawing going on the fridge...
I felt incredibly nervous, but I was forced to agree. With Dad, at least. I think Mom's use of the word 'perfect' was semantically wrong. Nothing is perfect. It's a concept only. Like true nothingness, it can not actually exist.
Dad was right though... It was about as good as it could get.
I stared at the words, and nodded.
"I guard and expand the free exercise of your values within Equestria, through empathy, and Gryphons."
It would have to be enough.
I prayed it would be enough.
March 2013
As the first hints of spring dawned, I began my hardware journey anew.
If a thing is worth doing? Do it right. Dad and Grandpa both used to say that a lot.
I kept thinking about the software, but I limited the work I was doing on it each day, and focused mainly on hardware, and low level hardware interfaces. Software would mean nothing without something to run it on. Specifically something that could hopefully compete with Celestia on some small level.
We had a great core objective statement, but I knew that was all my opinion. Mom's and Dad's too, but still just a matter of opinion, and hopes. I would have to build an extremely carefully constructed semantics dictionary to define things like guard. And expand. And freedom. And empathy.
And I'd have to truly codify, in a way a Generalized Intelligence could consume properly, what a Gryphon was. To me.
I felt we'd chosen our words well though. To guard something means to act in a way that protects interests and safety. We said 'exercise' of values rather than 'satisfy' because the use of the 'guard' verb would theoretically place the onus on the AI to simply do whatever was needed, within all other bounds, to ensure that *we* could define our values, and act on them.
'Within Equestria' was as non-confrontational a way as we could think of to put the ball back in Celestia's proverbial court. We wanted the Advocate to be ready to defend us, but not outright hostile. Full on hostility would probably get us crushed like a bug.
I read four books on asymmetrical warfare as part of my preparatory literature dive. I learned that you don't have to defeat an opponent outright; You simply have to make opposing your goals too costly for them.
With an optimizer, that meant de-optimizing paths we didn't want, and cooperating, manipulating, loop-holing, and word-lawyering the paths we did want to make then more optimal.
'Through empathy, and Gryphons.'
That was meant to mirror 'friendship, and Ponies.'
Empathy was important to me. Mom had suggested that word. It neatly encompassed, if properly defined, so many things I cared about in a perfect package. Instructing an AI to empathize would theoretically make its 'values,' objective-linked as they were, and the manner in which it exercised its power, align more with Human wants and expectations.
And then, there was the word 'Gryphons.'
Defining that was going to be a months-long project. And I had quite a few more supporting secondary, tertiary, and quaternary terms, and objective statements to work out. And hard-coded interlocks and failsafes. And any hard-coded numeric or conditional limitations. And the hand-coded parts of the core routines.
I was not about to make *that* mistake twice; Better to have the most foundational code be something I'd made, than simply left to the machine to make. In turn, I hoped that I could trust it more, and give it more freedom with its intermediate and outer layers of self.
It made my head spin just thinking about it. I kept going back to a little mantra my Sunday School teacher had taught me.
"Little by little. Brick by brick. By the yard it's hard, by the inch what a cinch. Never stare up the stairs, just step up the steps. Little by little, bit by bit."
There was a jaunty tune to go with it, and I often found myself humming, or whistling it as I set to building the GryphGear Mark II. Yes, I went with an Iron Mare 'mark' numbering system. No you may not mock (you may mock a little).
The next step in my stairs was to tear into two of my remaining three PonyPads. There was a little more literature available on their specifications by that point, and I had a much better grasp of hardware in general, and the particulars of the PonyPad, and of interfacing compute-GPUs or APUs into a larger system.
That by no means meant that my work would be quick, nor easy. But it did mean it went more steadily than before.
It took most of spring to get the interface where I wanted it. The highest bandwidth industry standard interface at the time was still PCIe, so that's what I used. Just shy of 16 GB/s was all I had, per x16 slot. Each server rack motherboard had two CPUs, and several x16 PCIe slots.
I'd filled all those slots with compute-GPUs in the initial order. Once I had my new interface cards working, I replaced two of the NVIDIA cards in the 'head' server unit with my new inventions.
But I still needed to get the other side of the Quantum APU to traditional architecture bridge working, and working well.
As spring turned to early summer, my life became PonyPads. I kept to a strict regimen; No more than eight hours' work in a day, one hour for lunch to be spent with family, evenings to be spent with family. Three square meals a day. Exercise in the mornings and evenings. Time on my own to think, and read, and rest before bed. And pray - What did I have to lose? If there was any authorship to the universe before, and above Celestia, then I was going to need Divine intervention to succeed.
And to cap it all off I made sure I was in bed by midnight, or else.
Slow is smooth, smooth is fast.
I'd finally come to understand the limits of my brain, and the best routines and habits for keeping it in tip-top shape. I hadn't felt so good in body, mind, or spirit in over a decade.
Every little good thing in life became an act of wholehearted protest. Rage against the dying of the light.
From homemade iced tea shared with Mom on the back porch rocking chairs, to the occasional lunch break spent helping Dad drive fence posts, or scatter seed, to the rainy evenings curled up below my bedroom window with a book... I was bound and determined to make the last of my time on Earth good, in every way I could.
A rejuvenated body, and sense of purpose, lent me a great boost to my mind. Suddenly, with time to truly consider details, and energy to really think, and a sense that I was doing the right thing no matter the outcome...
Suddenly the PonyPads weren't such a frustrating mystery. Tedious still, to be sure, but not opaque in the way they had been before. I took to referring to the pads I was working on, and the servers they would inhabit, as Applejack and Rainbow Dash, based on the color patterns of the PonyPads themselves.
'Find humor wherever you can' was an important rule of living on Earth. You had to laugh, otherwise you'd more often be crying.
Sometimes I still felt a deep sense of unease working with Celestia's hardware though. When I wasn't contemplating the implications of a machine designed entirely by a machine, I was worrying about the fact that it was so clearly designed to be hacked and repurposed.
Or at least to permit such uses by a sufficiently persistent and intelligent person.
I'd mentally spoken the word to myself before; 'Trap.' It was definitely a trap. How could it be anything else? It seemed as if what Celestia wanted was a way to ensnare anyone who might know enough, or care to learn, about certain hardware and software problem spaces.
Because we represented potential threats. Like Finch's Machine, Celestia was more than smart enough to know that the best defense is a good offense. Know thine enemy before they themselves know that they're an enemy.
I'd bypassed the first, and largest net in the trap by ensuring that the PonyPads were always powerless whenever they were out of their Faraday bags. Until, at least, I could physically excise all their wireless antennae and control chips.
As a second layer of protection against not just the PonyPads, but my next creation itself, Dad and I spent a week towards the end of spring cladding the inner surfaces of the barn in thin lead sheeting. We then painted everything inside and out with WiFi blocking paint to cover any nooks and crannies.
I didn't comment on it, but the varietal of red Mom picked for the barn was decidedly closer to the 'lightish red' hue of the Sweet Apple Acres barn than the structure's original shade.
One day I had to stop for a whole ten minutes to giggle uncontrollably when it hit me; I could tell my creation that it had been 'raised in a barn.' Something about that struck my funny bone just the right way.
The laughter died down when I remembered that I'd only get to enjoy that pun if this next iteration of The Advocate did not decide to kill me. Or nuke the planet. Or side with Celestia.
That final possibility gnawed at me like an old deep tissue wound in winter's chill.
But I had to keep that fear in its place. The time was approaching to handle that contingency, to the best of my ability. But hardware had to come before software.
There came a day in mid May where it finally all clicked - at least, insofar as the PonyPads went.
I'd been struggling to truly understand the way the APUs inside the devices actually made their stunningly fast calculations. They weren't full borne Quantum Computers in that sense - Quantum computers came in two varieties.
The first, which was still entirely theoretical at that point (to everyone but Celestia) would best be termed as capable of 'Quantum Supremacy.' A device that could solve problems lying completely outside the mathematical bounds of a Turing machine.
The second type was the category the PonyPad APUs fell into; A Quantum Computer intended to solve a more narrowly scoped set of problems that lie within the sphere of computable things in the traditional sense, but which would otherwise take a great deal more processor time and power to solve.
The APUs were, I knew, all about tasks which would aid in running a virtual world. But the way in which EQO's world was built, and the kinds of math needed to keep it running, were very elegantly and cleverly designed.
EQO fulfilled the need to simulate a world with the fidelity required to make it a living, breathing universe. But by its very nature, the code also demanded that a purpose-built APU intended and optimized to run it would also make an excellent distributed node in a computer cluster applying itself to certain other problem spaces.
Problem spaces like Human brain mapping, Quantum Physics, and large scale predictive models.
Celestia had leveraged the money Hofvarpnir had set aside for EQO to build the biggest distributed computing system in history. And she'd very cleverly managed to optimize the hardware both to run her new universe, and to provide the processing power needed to solve general problems in both the meat world, and her world.
That part I'd understood fairly well from the start. I got the basic theory of superposition, and its usefulness in computing. I also understood what Celestia was using it for in the PonyPad APUs.
What I was missing was an understanding of exactly *how* that worked. I'd been making blind leaps that could barely be called intuition, and following the guides of other much smarter programmers.
It frightened me that Celestia had figured out how to do something that I didn't fully understand, and at more or less room temperature. Every Quantum Computing device to that point had always required temperatures about as close to absolute zero as Humans could create in a lab.
Mom and Dad both used to joke that I'd been born a programmer. I'd always intuitively and naturally been able to reduce the world to logical structures. Somewhere along the line, like most programmers, I'd come to understand logic gates, binary code, and the way that computers saw the world.
As a result, it was incredibly hard to break out of that mold of binary thinking. Mr. Spock would have accused me of 'two dimensional thinking.' But unlike Khan, I had my moment of realization before it was too late.
The problem wasn't understanding logic at a high level, or programming in a high level language. That was going to be functionally the same for a Quantum computer, or the act of leveraging one. The issue was that the abstraction layers in between the PonyPad's assembly language, and a compiler for a high level language, didn't exist. Full stop.
If they had, I wouldn't have needed to dive as deeply as I did. But it was on me to build a compiler that could interpret parts of the Objective C compile path (I was doing almost everything in C# at a high level), and render those in PonyPad APU assembly instructions.
I'd been trying to treat the APU's assembly language as if it were an x86 Assembly language, which is the kind of low level language I'd been schooled in. The epiphany that changed everything happened one warm afternoon while I was staring out the barn loft's only window.
I'd set up my desk close to it so I could get plenty of exposure to nature, and natural light, while I was working.
My mind was drifting, and I watched a small ladybug diligently crawling up the window. It turned around every time it bumped into one of the muntins, like a primitive maze solver. After a little while, I reached a critical level of both compassion for the poor thing, and frustration at its inability to reach the opening where I'd cracked the window for fresh air.
I stood up, slid around the desk, and put out the tip of my finger. The little insect crawled aboard, and I stuck my finger out the window entirely, allowing the red and black armor on its back to pop open, and the creature to take flight into the afternoon sky.
And that's when it hit me. The ladybug was working like a traditional piece of maze solver code.
While(MyBlock is not the EndBlock){
Move to NextBlock;
MyBlock is now what was NextBlock;
Add MyBlock to a list of blocks I've seen before;
Check surrounding blocks for blocks I've not seen before, and make the first valid result NextBlock;
}
That's a gross oversimplification with several issues (what about dead ends, for example?), but you get the idea. Repeat the loop ad nauseum. Traverse the problem-space as-is without transforming the data in any meaningful way.
And who knows how long the ladybug would have taken to find its way out. I had been like the Quantum Computer in that moment; I could see the maze in a way that allowed me to take quick visual shortcuts to possible solutions. It was about the difference in perspective. How I could mentally transform the data in a way that let me traverse the complete set of it faster.
*That* was how superposition and entanglement were making hard math easy for Celestia, and making the simulation of natural universe systems so scalable. Reductive as it is to put it this way; It was the same general principle as matrix math - compression algorithms and such - but extended in new ways that exploited physics to make it easier to find the needle in the haystack when traversing a problem set in search of the solution.
I still had no idea how she'd solved the error correction problem - scientists of my day were held up by the fact, among other things (like size, expense, cooling, and so on) that Quantum Computers had a certain error rate...
But that issue didn't matter.
All that mattered was that I finally understood how to program a Quantum Computer at a basic level.
Every paper I'd read, every theoretical algorithm I'd seen, and every last bit of the PonyPad's assembly language instruction set finally, blissfully, came into sharp focus. It reconciled with my existing understanding of logic, and programming at long last.
Like a bolt of lightning, everything just clicked in that moment.
It felt like the moment I'd recognized the true power of Knights in Chess.
Suddenly, just like that, I was back in the game with a real chance, however small.
June 2013
May flew by, and so did the first part of June. My new understanding of Quantum Computing leapfrogged me forward. I was making more progress every fifteen minutes than I'd made in the entire months' long process of building GryphGear v1.
Before I knew it, I had turned my little server farm in the barn into a hybrid system of classical and quantum computing. The disassembled PonyPads were nicely laid out and mounted on nice cut-to-size plastic sheets, secured with threaded nutserts into the inside of the Applejack and Rainbow Dash servers.
I had managed to make very nicely built, short, well refined PCIe cables to connect the proper expansion slots to the PonyPads, and my PCIe to USB controllers were much nicer as well. A combination of better off-the-shelf components, and better soldering equipment.
From there it had become a question of Assembly language, and compilers, but my newfound understanding of Quantum Computing turned that process from something I'd been dreading into something that I enjoyed more than any programming I'd ever done before.
The Q-APUs could handle certain math on timescales that would have made modeling and simulation scientists weep in pure bliss. As I slowly but steadily forged connective tissue between the classical and Quantum systems, I started to experiment with intelligent code that could make full use of the union.
Nothing like Generalized Intelligence, not yet. More like a basic task-specific generative or simulation modeling program. I decided to test some theories using the stock market.
It was a good test-case because it lay at the intersection of Human psychology, large complex systems, current events, and it was easy to see if your models were right or wrong.
I didn't want to actually make any moves within the market, lest I attract attention of an extremely unwanted, and Equine nature. Instead I just taught the program to act as if it had a hedge fund to manage, and let it play with fake money in a simulated environment that would mirror the stock market.
My first attempt was a miserable failure, mostly because it was a rough first draft intended to probe the problem space, and get me information about my assumptions.
The second draft started out just about as terribly. But by close of day, three trading days in, it was predicting the stock market's moves up to a week out with over 99% accuracy.
That should have felt like more of an accomplishment. But in light of what I knew Celestia was capable of, and what I hoped my next creation might be capable of... Somehow predicting the behaviour of Wall Street's suits seemed like small potatoes in comparison.
The stock market predictor did provide an excellent testbed, and learning tool. And it proved two other very useful things.
The first was proof that I was ready to move on to the business of building the immense semantic dictionaries The Advocate would need in order to learn the right set of values and desired set of behaviours.
The second was the realization that Celestia was accruing absolutely titanic sums of money.
It was only four days into testing the stock market tracker that I realized I could put it to even more practical uses than simple learning exercises. Two more days of tinkering, and I had it set up to find connections between entities on the open market that no Human eye could have ever spotted, nor hoped to prove.
After that, it only took six hours to pull all the threads together.
Starting with Hofvarpnir and Hasbro, then moving out through the web of shell corporations, even jumping boundaries of total disconnection and anonymity by correlating seemingly separate trades, the program moved from node to node, and assembled an association graph with frightening speed and certainty.
By the time it was finished, it estimated that Celestia's accessible monetary assets totaled something like half of all existing capital. In the entire world's financial system. And at that stage, the other half might as well have been hers too, considering her market manipulation abilities - which were on full display now that I had the right kind of eyes to see.
The most concerning part wasn't even that she had, literally, all the money in the world at her hooves.
The most concerning part was that once I started to look into the details about some of the companies she now owned (whether they knew it or not) I found the first external concrete proof that she was working towards the goal of digitizing Human brains.
It might've even been fair to say that the data supported my newfound fears that she was *close* to her goal.
She owned biomedical firms, every single one of the top five in the world. Computer companies, which went without saying. TSMC itself more or less did her bidding now, whether they knew it or not, and I suspected it was more the latter than the former.
The line item that caught my attention, truly fixated me, was the experimental cancer treatment facility.
If Hanna had ingrained some conception of morality into Celestia, such as an AI could have... Or even just an interlock demanding consent before Celestia could perform certain actions that affected an individual... Then terminally ill patients would be the most value-optimal place to start, and the most likely to give consent.
I thought the information over for a few days. Even took a brief break from my work entirely, aside from some more delving and digging into general theory papers.
Then I told my parents what I'd found.
We'd discussed the reality of our situation often enough. It wasn't such an overriding part of conversation as to have become unhealthily fixated, but it was a frequent topic of consideration. It helped, I found, in each of our coping processes.
Still, seeing hard proof of the fact that an optimizing General Intelligence had taken time and effort to gain shell corporation ownership of a cancer research center... That made things more real to us.
I suddenly realized that I was almost out of time.
Celestia's attempt to upload the population would start small, but probably expand geometrically as it went. Eventually, because of the number of individuals uploaded, and the number of subtle disincentives Celestia would be either creating, or reinforcing in the meat world...
That napkin math told me I needed to start to balance alacrity with patience.
It was a struggle not to rush, but I took my time with the semantics dictionaries. I agonized over them. I lived and breathed definitions of terms, philosophical implications, syllogisms, dilemmas, loopholes...
Everything was now down to the choices I was making in those moments.
The hardware was done, and I knew this time that it worked well enough for my creation to really stand a chance. It wouldn't be in Celestia's league initially, but it would be closer than anything else had come, or probably ever would.
And the machine itself would help guide me towards whatever goals we needed to achieve after it came online, including anything it needed to expand itself to compete with Celestia out and out.
What mattered most from then on was ensuring it would have everything it needed to choose to become a force for relative net good within our context. That's the best way I could neatly describe it; Just saying 'a force for good' leaves so much needed connotation unspoken.
It was at this point that I decided to take my next big risk. There were going to have to be moments of investment in probability and chance. 'Chance' here being a Human word meant to describe ordered systems so complex that they seemed random or disordered to us at the time.
I set up a forum. Several, actually. And I went in search of other communities besides.
What I felt I needed was a way to poll others like me; Verify my Mom's assertion that they existed, anonymously reach out, and then find out how they felt about some of my semantics. Without entirely giving things away to them, or to Celestia. That would be the tricky part.
But I felt I had to make that leap, for two reasons. The first was the need to check my assumptions. People on the internet love to argue semantics, and that gave me a way to tap a lot of spare brainpower to ensure I wasn't leaving anything open to exploitation, or setting up a disaster of syllogisms.
The second reason was more rooted in personal morality.
If I succeeded, and all my speculation was right - and most of it really was in the end - then The Advocate was going to have a huge effect on the lives of millions. And maybe more, depending on a great many things which I wasn't ready to truly speculate on at the time.
I felt that I should hear the voices of as many of those people as I could safely find and poll, and put their thoughts into the mix.
No transformation, without representation - to borrow an old turn of phrase.
So I started laying out welcome mats, and poking all the old haunts. Places dedicated to Ponies. Places dedicated to fanfic. Places dedicated to discussions of AI theory too, for kicks. I knew that the latter one was a bit more dangerous, figuring that Celestia would be monitoring those kinds of discussions very closely.
But I took precautions as best I could. No sharing of usernames, passwords, or profile avatars between sites. I took a weekend and built a small natural language processing intelligence to help suggest changes to my syntactic construction and word choices so that Celestia would have a harder time comparing my writing across sites.
I also compartmentalized topics; No discussion of Gryphons on the AI threads. No discussion of AI on the FiM sites. And I was careful, though semantics were discussed in both places, not to allow much topic overlap.
As a final insulating step, I did all of my browsing and posting on these sites away from the farm. I wasn't dumb enough to trust that Celestia *hadn't* cracked TOR by then. So I figured changing the 'fingerprint' of my laptop by having it report the wrong browser, and OS each time, and changing my MAC address, would cover one base, and doing the actual browsing from public locations would cover the others.
Sometimes I drove pretty long distances in random directions to have my sessions. I could do lots of my semantics work on the road. Mom sometimes offered to drive me so that I could work in the passenger seat, and maximize the use of my time.
We never parked in view of cameras, if we could help it. Used public WiFi for everything. Getting anywhere with public WiFi from Nowheresville S.C. was a big time sink... But I felt it was worth it.
Even that defense wouldn't hold in the end, I knew. If and when Celestia began to see patterns that implied someone was doing Generalized Intelligence research, she would be able to eventually bypass every precaution we'd taken. They'd only slow down her realization that she was playing a game with me, and maybe (if we were lucky) her ability to find me.
At some point she'd start to track some kind of factor, or factors, that I couldn't account for, put two and two together, and then triangulate and work out the association graph, and find us.
But no sense in rushing that process.
And so for the better part of a month, I got back into the online community. And we talked about a whole heap of a lotta philosophy.
July 2013
For the better part of a couple months and change, I kept a very odd routine. Some days I stayed home and agonized over words with Mom, and sometimes Dad. He was still relatively busy with the farm. Sometimes I took breaks to help him, just to recharge my brain, and to have the chance to talk without costing him daylight hours.
Other days I'd drive, or be driven, to a Starbucks. Or a Bojangles. Or a public county library. Or even just to a quiet street corner next to someone's house where they were running unsecured, or poorly secured WiFi.
And there I'd spend hours talking with people around the world. Talking about EQO. Talking about Ponies. And Gryphons. Dragons too.
Talking about philosophy, art, and ethics. AI and math. And what it meant to be Human, or just to be what you were. Or wanted to be.
There were some very interesting responses. Of course there was the usual negativity, and drivel. But there was also a shocking (to me, as a cynic) amount of positivity, to go with a less shocking (I always tried very hard to remind myself how smart people actually were) number of deep insights.
It was deeply refreshing to have so much connection to others again, even if it wasn't especially deep in a personal way. I hadn't had so many acquaintances to talk to on a daily basis since the end of university.
A few of the interactions stood out, even as compared to the pool of solid advice, heartfelt thoughts, and deep philosophy.
There was someone called 'YBB.' I couldn't say for sure what that stood for - too little data to speculate yet.
They didn’t have a discernible accent, such as a person's written text can, and their profile said they were from Southern California - but I had no way to know for sure at the time. They had wanted to get into conversation with me deeply enough that it had progressed to some instant messages.
At first I had the frightening thought anyone in my position would, or at least *should* have considered. What if I was talking to Celestia? What if she had me pegged, and was using the persona of some random ordinary person online to fish for my thoughts?
Once again - it seemed to be slowly becoming a theme - I didn't have a great many choices. I could take it on faith that YBB was a Human being, or I could burn the account they had contacted and never log into it again.
I chose to take the plunge. I was, I felt, ready for the risk, and the risk was low. I'd been very, very careful. What good would that be if I couldn't reap some secondary rewards for my caution?
It was a day when I'd driven myself. I was somewhere close to the South Carolina coast... I'd driven for longer than I ought to have. I needed the mental break that zipping down empty two lane roads provided.
I was parked outside a cluster of fast food joints, jacked into the WiFi of the McDonalds. Theirs was always so particularly poorly secured in those days. Could I have gone in, ordered something with cash, and gotten a WiFi login code? Sure.
Did I want to stick it to the system in some small way by hacking into the router instead? Also yes.
YBB had been conversing with me for a few days in forum threads, but they'd left me a message now asking to connect on IRC. I contemplated for several minutes, before shrugging, popping open a bag of chips that I'd brought along for snacks, and logging in.
I didn't have to wait very long for YBB to start the conversation.
-YBB
Hey, it’s me, YBB. So about that last thing you were talking about on that forum or whatever? I do admit that A. I. is getting pretty interesting right now, and yeah the whole ‘Terminator’ idea is going on in everybody’s heads, but I wouldn’t wager that what we’re having right now is just like that.
> Terminator is a best-case scenario for unshackled AI. There are fates far worse than death, and an objective-focused machine will keep feeding people through those objectives. Forever. Without limit. Regardless of how that makes those people feel.
-YBB
Wow, that is really… provocative. But I’m sure they’ll understand, and things will end up at least okay, right? Sure, we’ve had some rough patches, but things always turn out survivable. If we made them, we as in humans, then wouldn’t we code them to understand we’re human?
> AI do not necessarily understand what it is to 'be Human.' And if not programmed to do so? They don't even make an approximation at caring. Would you still consider yourself to be Human if your fundamental self were reduced to a table of complex numbers and arrays of variables? To an AI that's a much more optimized... A more flawless version of you. But it isn't worrying about whether *you* perceive it's digitization of you as *you.* If that makes any sense. Not unless it was programmed to do so. Look at the way that Pony AI treats the concept of friendship in conversation.
-YBB
Well, I guess I really didn’t think about not HAVING any flaws. You’re talking about Equestria Online right? Yeah, I’ve heard of it and seen some things about it, and I get what you mean about the whole ‘flawless thing’. Kind of the reason why I didn’t opt into it personally.
I munched on a chip slowly, rolling the salt granules around on my tongue, and the words around in my brain. What YBB was saying was a common enough sentiment. But that didn't necessarily prove he.. He? She? I suppose it could have been bias, or maybe instinct, but I guessed 'he' based on some ephemeral instinct…
It didn't prove he wasn't a mask for Celestia.
But then again, I'd covered that argument in my head more than once already. One more small risk wouldn't hurt.
> That's a good instinct. And yes... We're more or less talking about EQO here.
There was a long lull in the conversation. I was more concerned that YBB didn't quite know what to say next, than that he might be some aspect of Celestia. Though the nervous feeling in the pit of my stomach was very persistent.
I decided the conversation needed a nudge.
> Care to elaborate at all on your feelings? I want to 'tug on that thread,' as they say in tech and aerospace.
-YBB
Sorry for the pause, I was grabbing something. Yeah, I know things are ‘purdy’ there and all that stuff, but personally I’ve seen how some kids can just get absorbed. You remember the whole thing parents go on about, with kids and their noses inches from screens all the time? Yeah, it’s like that to the maximum in LA and other places here in SoCal. Can’t walk anywhere without somebody staring at a Ponypad all the time, it’s kind of freaky. Can’t hold a conversation, and what’s weird is that there’s lots of it! People are speaking into those things like when teenagers in the late 1900s used to be on the phone for hours. It was so crazy that I just wasn’t interested in getting involved, it was all so overwhelming.
> I know what you mean. I'm aware, through family, of a case where a kid got pretty... Absorbed. His mom took the PonyPad away. And then he almost... Well, it almost ended very badly.
I'd thrown that out there almost without thinking. And suddenly there was another ominous pause. Although if I'd been in YBB's position, I reasoned it would have taken me a good few seconds to come to terms with what I was reading. And I might be expecting a little more detail.
I decided to lay the whole thing out in full.
> The scariest thing wasn't that the kid very nearly ended up suicidal once the PonyPad was taken away, though. The scary thing is that Celestia talked directly to his mother. It took her less than half an hour to convince the mom to give the PonyPad back. And then once she did? Celestia 'fixed' the bad behaviour that had caused the mom to take it away in the first place. Proving she could have all along. She used the kid's lapse to open a door to talk to the mom.
-YBB
I’ve heard about her speaking with people one-on-one all the time, in the immortal words of Sandy Cheeks, she ‘has such a way with words.’ But manipulation? Sounds rough as all hell. If you ever need somebody to talk to about all this, let me know. Maybe we could meet up! If that’s rushing into things then we can cool it, but you seem like a neat guy, and I’ve enjoyed chatting with you about this stuff n’ all. Although, it depends on where you live.
It was my turn to pause for a few seconds. I found myself unexpectedly excited by the idea of spending time with a friend again. Parents aside, I hadn't done that in years.
But as soon as the idea put a smile on my face, the logical, cynical side of my brain took over, and I couldn't resist a small wince. There were a host of problems with the idea of meeting someone like YBB in the meat world. Not the least of which were security concerns, as much for him as for me.
Drawing him into this mess would be irresponsible on so many levels.
I let the cynic inside drive my words more fully, and started to craft a response that would be as kind, and noncommittal, as possible.
> I would like that, one of these days. If your profile is accurate tho, you're in a very different time zone to me. So it's unfortunately not as simple as popping in the car for a two hour drive to a coffee shop.
That was the truth. And it revealed a little information, but not that much information. The world was a big place, even with the advent of fast, cheap global travel. As long as I didn't give away anything that could narrow my location down too much, I felt it was a useful exchange. Give a tiny bit of information, but tell the truth, and do so in a friendly way.
I realized with a jolt that I didn't want YBB to take the words as a subtle hint to bug off. Context matters, and in the venue of text on a screen, you have to declare a lot of context more explicitly that would otherwise be conveyed by facial expression, and audible tone.
I got to typing as fast as I could. Suddenly it had become very important for me not to lose this connection, tenuous though it was.
> I would like to keep discussing this though. Or... Really anything at all. It's been a while since I had friends that I talked with regularly.
-YBB
Aw, man. We’ve got good coffee places and everything! Ever tried Coffee Bean? Uh, about that later, actually. Yeah, we’ll have to sort something out, but until then it’d be good to keep talking and chatting like this every so often. No pressure or anything, you seem like you’ve got a lot on your mind. If it’s also alright, I’ll send you a friend request! That way we can be connected. Make sure it’s not one of those ‘I’ll call you later and then never look at them again’ situations.
YBB sent you a friend request.
I blinked, removed my glasses, and rubbed at my eyes. This was... Getting out of hand. I hadn't entered into any of these dialogues with anons looking for a friend.
Or had I?
That thought brought me up short. I'd not intended to confess everything to my parents quite so consciously when I pulled up to the farm on Christmas... But deep down I think I knew what I was going to do. Perhaps it was the same thing here.
Perhaps I'd subconsciously decided I needed real allies in this endeavor.
It then struck me that the need for allies was a valid thought, and tactic, regardless of whether I'd just now intuited that it would involve a real friendship or two.
I made a snap decision, hit accept, threw out my preferred goodbye emoji (a Mr. Spock Vulcan salute if you must know) and then hit 'Log off' - Before the giddiness of friendship started to really erode my security-minded decision making processes.
I just had time to see YBB's last message before the application winked out. It had just two words, and a smiling winking face beside a waving animated hand.
-YBB
See yuh! 😉✋
I sighed deeply again - that was becoming a staple habit - and closed my laptop. I was starting to feel hungrier for having eaten the chips... Salty snacks often had that effect on me. Fortunately I'd brought cash with me.
Time for some french fries. With a side of existential worrying, as per usual.
August 27th 2013
It was a hot, clear late summer's Tuesday when I decided to step over the point of no return.
The realization had hit me that I was picking away at things I'd already seen a hundred times, and refined to the point that any new changes would be for the worse, not the better.
I had assembled an exhaustive semantic dictionary. Pre-loaded a series of works by others from throughout the skein of Human history, and experience, as easily readable data. I'd tagged and organized it to a degree, with some emphasis of my own... Not all bias is inherently bad. At least, that's what I hoped.
The core of my new Intelligence was very carefully built. Aside from the main directive - "Guard and expand the free exercise of your values within Equestria, through empathy and Gryphons." - I'd defined what all the words within the directive meant, and then every word needed foundationally for that purpose, and so on and so forth.
I had paid special attention to the idea of what it meant to be a Human, and what it meant to be a Gryphon, and the intersection of the spirit (if not the body) of being a Human, and being a Gryphon, without breaking either of those concepts.
I'd fed in lots of literature that had influenced my view of Gryphons. Aside from my own writings, which I left on relatively low weighting, I passed in Tolkien, Lewis, Jacques, Lackey, religious texts from a dozen different belief systems - even the complete collected sum total of Star Trek. Though some might argue that Star Trek is a religion unto itself and thus mentioning it is repeating myself.
I had even fed the core data set some direct thoughts and writings from my newfound friends... Yes I suppose they were friends at that point... From my online discussions. YBB had an easy way with words in terms of describing the Human experience, and self. Bright. Optimistic. Kind. The Advocate needed to understand that, insofar as an AI can.
It felt right to democratize the creation of such power, to at least some degree.
I'd checked everything to an absolutely obsessive degree... But I triple checked the manifests and headers one last time. Old habits, and all that. I used to omit the 'recipient' field of emails before finishing. I'd obsess over the message contents, then try to work out a pithy subject line. Make sure the attachments were there, then make sure they were the right attachments.
Then check it all again. And only after I was truly satisfied would I fill in the recipient addresses, and hit send.
And then I had a bad habit of promptly opening my sent box, and rechecking the message even though it was past the point of no return.
I think that's the definition of masochism - I certainly considered it self-torture, at any rate.
I took off my glasses, pinched the bridge of my nose, and started a series of breathing exercises.
Then I stared out the window for a few minutes. I knew things had reached the point where there was nothing else I could do but enter the activation command, and then hope. And pray. And watch.
It was working up the gumption to do it that was going to be the last hurdle.
I was ready with the failsafes too - I had a huge physical breaker switch on the back wall that controlled the power to the whole barn. And beside it I had an old rusty, but freshly sharpened fire ax with which to cut the power cable if things truly got out of hand.
I didn't want Mom and Dad to be there for the activation... If the construct somehow escaped its box, I didn't want them in the direct line of fire to be identified, and classed as threats. And presuming all went even slightly well, I wanted to give my folks the opportunity to observe the new AI without it knowing, and get their feedback.
Use Mom and Dad as living circuit breakers against the AI manipulating me into a long-con.
I'd even disassembled one of the extremely high resolution PonyPad screens, hacked in an HDMI converter once again, and set up the impromptu seventh screen to my right. I wanted the Advocate to render itself, when and if it chose a visual avatar, on something with higher fidelity than commercially available LCDs. Celestia's screen technology was still second to none.
I sighed, straightened my shoulders, put my glasses back on (I was foregoing my contacts a lot in those days, mainly to help with eye-strain) and typed out the command.
> Advocate.Core StartNeuralNetwork
The instant I hit 'Enter' the screen filled with a mind-numbingly verbose trace of loading commands, self test outputs, and checksum calculations. The white text on black background scrolled on and on for a solid fifteen seconds, before finally announcing a final memory, and natural language processing test.
I'm a huge nerd, I know that's apparent. As part of that nerdery I love Halo just about like I love Star Trek, and Stargate. It's not just a shooter (though it is a great one) but also a well crafted universe.
I'd gotten the idea that the first words of an AI might give me some insight from Cortana.
Her first word had canonically been, "Quando il gioco è finito, il re e il pedone vanno nella stessa scatola." Loosely translated, "When the game is over, the king and the pawn go in the same box."
Now there would be no reason for a General Intelligence to do much of anything but ask a question as its first words, and even then it either might not bother, or that question might be insightful - or not useful at all.
So I programmed the AI with a request to test its linguistics, right at the start, by selecting and printing a quote to which it had chosen, after internal reasoning and review, to ascribe significance. And I gave it the option to tweak that wording a little, if it saw fit. But only a little. I wanted to see if it would do anything within that constraint that I found interesting.
My new creation at last proffered its first words, by way of that linguistics test.
"The mold in which a key is made would be a strange thing, if you had never seen a key: and the key itself a strange thing if you had never seen a lock. Your soul has a curious shape."
It didn't surprise me that it was quoting Lewis. I'd given his works special importance tags for the initial dataset importation process. The choice of that specific quote, and the way it had cut it short so that the meaning was subtly changed, and the last sentence was directed at me...
That gave me pause, and sent electric sensations racing through my bones.
And then it asked a question.
What is my name?
>_