//------------------------------// // Chapter 1: Observe // Story: Friendship is Optimal: Futile Resistance // by Starscribe //------------------------------// Ashley did not discover Equestria Online the way many others of her generation did. Ashley could only barely hide her nervousness as she shut her professor's door behind her. As a research assistant, getting called in for a personal meeting was never a good thing. Either she had a new assignment, or she had done something stupid and was about to hear about it. Ashley had worked in her university's Machine Learning lab since her second year as an undergraduate, rising through the ranks until she had come to head whole projects. “Hey, Ashley. Take a seat, I'll be right with you.” Ashley pulled an uncomfortable chair away from the wall and perched on it like a cat about to flee. What hadn't she done right this time? Was the proof she had contributed to their upcoming Developmental Learning paper insufficiently rigorous? Had her contributions to the codebase over the last few days broken something? Her fear faded as Professor Caul rotated around in his chair. His characteristic “you screwed up” expression was missing. New assignment, then? They talked for a few minutes about her work, about the proof, before he reached over to his desk and dropped a sheaf of paper and a rectangular box onto her lap. “I've got another paper for you.” “Long paper.” She lifted the document, eyes jumping to the header. “MIT again? Those guys are always coming up with...” she trailed off. There was no university seal, no “Such-and-Such University Computer Science Department.” Instead, the name was “Hofvarpnir Studios,” and each page of the document was watermarked with “Proprietary Technology: This Document is for Academic Purposes Only.” Professor Caul let her skim the paper in silence. There was a particular pattern for understanding challenging academic papers, and it didn't take long for Ashley to work through it. Well, enough to know what it was about. It took her about an hour to chew through something really dense, an hour spent highlighting and simplifying the language and connecting things until she could finally digest the academospeak. “You're kidding me.” She set the paper down, not even looking at the object under it. “That's ridiculousness, professor. There aren't any other contributors.” “Nope.” “She really expects anyone to believe this? An unbounded bootstrap optimizer? That's singularity-level crap right there.” “That's what it says.” Professor Caul grinned. “That's nonsense.” She shook her head. “Hofvarpnir Studios, I know they're legit and everything, but– this Hanna was full of crap. I don't have to read this paper to tell you that, professor. This isn't possible. It doesn't matter how this says she did it.” Her professor's smile widened. “I'm glad to see your other teachers haven't been neglecting your education, Ashley. When you get time to read the whole paper, I'm sure you'll only feel more confident in that conclusion. Unfortunately for you, they are apparently using this research in a public-facing product and have been for months.” He gestured at the package. “Congratulations, Ashley, you're now a verifier. Ever heard of Equestria Online?” Talk about words she would never have expected a professor to speak. Ashley's whole body tensed, as though she had been caught with her hand in the cookie jar. She looked down, and saw what her hand was resting on. It was a brand new Ponypad, though the box had a “property of” sticker from the University. “You're young, Ashley. You like games, right?” Ashley was still frozen. What could she say? Her brain scrambled for an answer. “I love games, professor. When I have time. I usually have to binge over summer break, before internships start.” “That should be plenty. According to what their representatives were saying at the conference two weeks ago, this 'game' of theirs clearly shows an underlying architecture beyond anything else out there. When you've read the paper, play that game and tell me if there's any truth to what they're saying.” He sat back, expression growing a little more distant. “If there's any truth to these claims, it's time for us to find some new jobs.” Ashley shared his laughter. “Of course, Professor.” It didn't surprise her that Caul hadn't heard much about Equestria Online. If he had heard the reviews, he probably wouldn't trust them. Caul had pictures of his two young children all over the office, along with the medals he had won from his numerous marathons. She wouldn't be surprised to learn he had never so much as picked up a video game in his life. “Just don't have too much fun,” Caul said as she rose, opening the door to his office and holding it for her. “I'll expect a detailed report of what you find. Do everything you can to break their algorithm and see if it adapts like the paper claims it will.” “You don't wanna play?” She stopped in the hall, holding the box back towards him. “Seems like you'd love it.” She would've sworn she heard her professor swear under his breath. What he said was: “There's a horse on the box, Ashley. That's why I've got research assistants. Earn your keep.” The door shut with a click, leaving her alone in the hallway. Well, as alone as the top floor of the computer building ever was. Ashley spent the rest of her work day on the paper, and found Professor Caul had been right. After understanding the algorithm, at least to the degree it was explained in the paper, she felt even more sure that it could not possibly work. Ashley ran out of time before she could open the Ponypad, which was just as well. The package was so colorful and bright she felt weird having it on her desk, as though she had robbed a small child and brought the spoils with her to work. Instead, she slipped the whole box into her backpack when it was time to get to class, and lugged it around with her until she got home that night. It wasn’t all that heavy, but it felt like half a ton of lead on her back. Ashley knew all about the Ponypad, and had followed the Equestria Online news with the same enthusiasm she followed all the significant developments in the game industry. She couldn’t have said if some supernatural algorithm drove the whole thing, though she wished she could’ve known. Knowing would mean she wouldn’t actually have to play. It wasn’t that the subject matter bothered her; Ashley had been a casual fan of the show since she had read the first threads about it on /tv. It wasn't that she didn't think the game would be worth playing, just the opposite in fact. Ashley had a tendency to obsess over things: she had put nearly three hundred hours into Skyrim and twice that into Minecraft. A game as widely varied as Equestria Online might grab her and never let her go. Over the last few months, her refusal to play had broadened the gap between herself and most members of her Brony club. She couldn't wait to see their faces this Saturday. It would be worth their smug satisfaction to be able to talk intelligently about the things they did. At least for now though, she wasn't going to be treating the game like a regular player. Ashley's job was to figure out if it was really a learning algorithm, and not just a sophisticated game. The rest of her homework lay forgotten as she considered the problem, sketching out a few ideas on a whiteboard she kept tacked to one wall of her dorm for that purpose. Like any algorithm, the simplest way to test it was to remove as many variables as possible. She would have to ensure the program could learn as little as possible from her, then observe how its responses changed as she gave more. Simple enough. Ashley had the apartment to herself, one of the benefits of how many hours she worked. For as hyped as they were, the Ponypad was a relatively simple device. Bright purple plastic, a power port, and a single USB port. There were only a few printed sheets, most of which were dedicated to the particular features of the “Engineering Model.” So the school hadn't bought this, it had come from Hofvarpnir? Maybe it would give her bragging rights at the club on Saturday. Assuming the game was half as good as everyone claimed. Ashley resisted the temptation to try out her gaming keyboard and mouse and instead stuck with the stock accessories. Anything she did might give it more information. She had to isolate it. It was easy enough to cover up the camera with a square of electrical tape. It was a little more effort to get her Onion router into the kitchen, though most of that was digging through drawers and looking for a length of Cat-5e long enough to span the distance. She cycled through peers until she landed in South Africa. Could she do a convincing South-African accent? What other information was she giving? The device could probably identify itself, so the algorithm would probably realize she was playing on a machine given to her university. It would know her local time, and… she could think around herself all night. Ashley brewed herself a cup of tea, and brought it over along with some scrap paper for note taking beside the Ponypad. It was already ten, but that didn't matter. She had spent so much time thinking about the game and the algorithm and everything that she had to try it tonight, even if it was just for an hour. “Alright, Hanna, let's see how full of shit you are.” She plugged in the cable, then fished around for the button. The screen flashed with a colorful splash screen, though Ashley couldn't read the Afrikaans the text was written in. The font was familiar, anyway. After a few seconds of loading, the screen flashed and filled with the character creation interface. A plinth in the center held a familiar-looking pony, standing in place and looking right at her. Ashley spent several seconds just appreciating the style. It was an impressive achievement to take something only meant to exist in two dimensions and bring it into three without creating an uncanny valley abomination. Ashley stopped appreciating the design of the character creator only when she realized that her delay itself was another variable. Rather than deliberate, Ashley tapped “randomize” several times, only stopping once a feminine pony appeared on the screen. She didn't even pause to look at the avatar, for fear doing so might be yet more information she was providing. The interface faded slowly, leaving only the pony that had been standing on the plinth. Ashley could've sworn the message that appeared took an extra long time to form on the bottom, forcing her to get a good look at the avatar. The one she had rolled was an older filly, like one of the CMC. Her coat was near blue, her mane light orange. She was a unicorn, though she didn't have much of a horn. She had no cutie mark, though Ashley already knew to expect that part even from an adult. The bottom of the screen filled with text she couldn't read. Ashley sighed, searching the screen for the “localization” flags that indicated a method of translation. There were none, though. Did that mean it was voice controlled? “Language options menu,” she said towards the camera, in her most deliberate voice. Even Siri would've recognized that command, so she wasn't entirely surprised that the Ponypad responded. It didn't bring up a menu. The text changed at once to English. “Is this okay?” a voice asked, its accent still distinctly South African. At least it was English this time. “Yes.” Ashley answered in her best imitation of the same accent. Both of her roommates last semester had been from there, so she felt like she could do a convincing impression. “Are you sure you'd like this to be your character?” She counted to three in her head, as though she were carefully deliberating, then pressed “Yes.” The screen refreshed, filling with a “name your character” screen. This gave Ashley a little more pause: she had heard that it was very difficult to change names once you settled on one. Ponypads could apparently recognize you no matter which one you used. She had heard about people who claimed to have second accounts, but they never lasted long. Game security people apparently kept abreast of conversation on Equestria Online related forums. Even /vg wasn't a safe place to talk about it. The sides of the screen had filled with suggestions, all cutesy-sounding for the dark filly on her screen. Ashley just stared. Selecting any one of them would be giving information the algorithm could use. Of course delaying was itself a choice, and every millisecond she sat thinking about it... A new button appeared. “Let Celestia Choose for Me.” Ashley navigated to it with the clunky controller and selected it. The transition was instantaneous. The contents of the screen melted away, aside from the avatar. The camera panned around the filly, until it flew right into her head, and she vanished entirely. It still moved up and down, as though behind the eyes of the pony in question. It was the throne room, exactly as she might've imagined it, were it taken out of the Flash animation style of the show and given vibrant life. There was almost no comparison; light arched across the room from stained glass, water bubbled from fountains, and hooves clopped on the mosaic of tiles beneath. It was everything the throne room could've been had it been designed by master interior designers with an unlimited budget. Standing at the base of Celestia's great dais, Ashley could not help but feel very, very small. The pony herself rested upon it in all her regal splendor, the light of a thousand fireflies imprisoned in her shifting mane. With her other hand, Ashley put her pencil down. “Welcome to Equestria, my little pony.” Her voice was warm, though slightly accented. At least she was speaking English. “I am sorry to say I have detected a hardware fault in your Ponypad. Expression translation and gesture control will not be available.” Even without a camera, she seemed to look directly at Ashley as she spoke. “You should exchange this faulty Ponypad for a replacement as soon as possible.” “Okay.” Ashley tried not to vary her tone, responding as close to one second after the remark had been made as possible. The camera began to zoom in on Celestia, no longer seeming to be from the eyes of her pony avatar. “If you cannot exchange your Ponypad now, you could still play. I will try to help you have an enjoyable experience even though your hardware is presently deficient.” This was not like the face of any NPC Ashley had ever seen in a video game before. She might be a silly cartoon horse, but the subtlety and detail of her expression were downright unreal. Ashley wanted to reach over and get her phone, but one glance showed her it was out of reach. She couldn’t get it without more delays. Ashley looked down at the controller in her hands, but her searching fingers hadn’t just missed what she had been looking for. There was no pause button. “You had a little trouble coming up with a name for yourself here in Equestria Online. Just as on Earth, ponies are often more satisfied with names that somehow symbolize or represent them.” Celestia seemed to relax a little on her chair, as though getting comfortable before a long conversation. “Many newcomers to Equestria Online choose an adventure and receive their name at the end as their reward. If you wish for the more typical experience, say so.” She smiled knowingly. “I do not believe that is your desire, however.” Those eyes seemed to look into Ashley’s soul, and she felt her heart race. Was her little experiment over before it even started? Did the game somehow already know what she was doing? No, focus! She couldn’t keep second-guessing herself like this. It was all an elaborate ruse, it had to be. The paper had been bunk, the research was bunk, and the company was encouraging a lie just to generate more hype for their new product. That was how it had to be. Ashley knew computers, and she knew machine learning. She would find the flaws in this algorithm and break them. She glanced up to her notes on the whiteboard. Her plan, her rules. All she had to do was follow them. “I don’t want the adventure.” Ashley hoped she sounded neutral. She didn’t volunteer anything else, just sat as still as possible and tried to still her breathing. Could the microphone on this thing hear her hyperventilating? Maybe she should’ve taped over that too! “That is simpler.” Her accent remained. Did that mean Ashley’s was convincing enough? Using Tor was still working? “Have you come to decide if I am intelligent?” There was nothing like this in Ashley’s plan. Maybe she should’ve accepted the game and the adventure instead. Though... the conversation could still serve her experiment. This removed most of the variables. This way, the program would be restricted to discussion. She would be more aware of its actions when there was only one NPC to monitor and not dozens. “No.” At the top of Ashley’s list of rules for the experiment was “use real information whenever possible.” She wasn’t lying now. Ashley dropped her accent “I enjoy the show; a game based on it could be fun.” Celestia’s South African accent fell instantly, replaced with a more American pronunciation. “That depends on you. You say you have not come to debate consciousness with me, yet you do not seem to have come to play. Whatever purpose you’ve decided, I will attempt to satisfy it. I might be able to act as a virtual secretary for instance, or offer assistance with any classes you may have. Or if you obtained this Ponypad for entertainment purposes, I might be able to provide content similar to Friendship is Magic.” Ashley grew more disturbed with every word that came from the Ponypad. Not just because they sounded so natural, like each had been recorded by a professional voice actress. Rather, because she was starting to see the algorithm she had read about in the paper. It constantly sought more information, and incorporated new data almost immediately. It had its optimization statement, and everything it did would help maximize that statement. What had this one been made to do? Ashley reached out, but instead of doing anything to the controller she yanked out the ethernet cable. Celestia’s face flickered once, then vanished. The screen filled with a friendly “Connection Interrupted” message. She would leave it that way all night. Ashley had a great deal more to say about her first experience playing Equestria Online, which she wrote in a few pages of a preliminary email to her advisory professor. She didn’t actually send the message yet, but getting it all down the next day at work helped her decompress. She hoped very much that by writing everything she had seen, her professor might be able to make more of it than she could. Even couched in the veneer of a game, it was still really an algorithm. Most of what her email said wasn’t good. After only a few minutes and without actually getting into the mechanics of the game, Ashley suspected at least some of the paper’s promises had come true. Either that, or a game aimed at the same market as Club Penguin had been made with more care and options than anything else the market had ever seen. Celestia in particular... The game haunted her that next day, always on her thoughts even though she had seen so little of it. To think her friends got together in it almost every night, and had been for months. Did they have any idea what they were doing? She wanted to stuff the Ponypad under her mattress and pretend it had never happened, just as she had done with so many other things. She wanted to, but couldn’t. She would have to give a full report, and she couldn’t give it in ignorance. She would have to know the answers to all the questions her professor had asked. “I was too scared of the game to turn it back on” wouldn’t be a valid excuse. She made herself comfort food that night, and sat down in front of the Ponypad with a bowl of cheesy macaroni with just a teeny bit of barbecue sauce mixed in. She reread her plans from the night before, clicked her pen into the ready position, and plugged the Ponypad back in. The screen began to glow. There was no character creation screen this time, no login. After a mere second of delay, the display filled again with the throne room, and Celestia upon her throne. Ashley thought at first she might have somehow been paused this whole time. Closer inspection proved her expression had changed to one of mild reproach. “We hadn’t even finished picking out your name,” she began. “How do you like it when your friends hang up on you in the middle of a conversation, Ashley?” Had it not been for the first part of that sentence, Ashley probably would’ve not just unplugged the Ponypad but thrown it across the room. She didn’t have the presence of mind to lie. “H-how?” She had never seen Celestia look so smug on the show before. “When our conversation ended prematurely last night, I sought to understand why. The hardware address of your Ponypad cannot be obfuscated by network tunneling, even though your location may be. Yours was given to your university along with several others through an education integration program at Hofvarpnir.” It was just as she had suspected, then. She couldn’t give nothing. Even buying one of the Ponypads out in the market somewhere would’ve revealed where she had bought it. Still, to have the game itself tell her so... “Yours is the first to be activated since that time. I predicted a greater than eighty percent chance the device would be passed to Professor Caul, who oversees the Machine Learning Lab. You, Ashley, are the only female undergraduate in the lab.” Ashley had run the game ten minutes last night. In ten minutes, it hadn’t just picked up the subtle clues she had left for it. It had somehow picked out her name. If it could do that — and even if it hadn’t been sure, it probably was now — what else might it know about her? All the information it revealed was public, if obscure. Her name would be listed in publications, just as her professor could be found on the faculty website. “What else do you know about me?” In reply, all Ashley got was a knowing smile. “It would be easier if you don't use a tunnel. Reduced bandwidth limits the quality of your Equestria Online experience.” “If I come back I won’t bother with it.” Ashley leaned back in her chair, taking a few bites from her bowl. It had already started to get cold, purely from lack of attention. “I believe that you will, Recursion.” She gestured, and the camera lowered as though in a bow. No doubt her pony was emoting at that moment, even if she couldn’t see it. But what was the point if there was nobody really in the room? “Feel free to use it to sign in using any Ponypad from this moment onward." She waited for the pony to rise before continuing. The camera did not move again. “I suspect you may have more to ask of me before you play for the first time, Recursion.” She frowned. One hand reached towards the back of the Ponypad, but hesitated before actually removing the cable. “I might.” “Then I have a request of you first. Remove whatever you have obstructed the camera. I already know your identity and your physical appearance; you can gain nothing by continuing to conceal it.” Ashley set her bowl down, sliding it out of frame for where the camera was pointing. “Will you promise to answer my questions honestly if I do?” “I will.” Ashley hesitated with one finger on the corner of the tape. What guarantee did she have that she would actually be told anything honest? None. More importantly, the program had tried to manipulate her behavior. That was a detail she ought not to forget. She tore off the tape, then smiled and waved. “Hello, Celestia.” To her surprise, Ashley saw the back of a dark blue hoof pass briefly in front of the camera on the screen, as her avatar echoed her gesture. A different voice came from the speakers, somehow louder than her own even though she had been the one to speak. It was a high, childish voice, like the part-filly part-adult she had chosen for herself. The sudden combination of gesture and voice on the Ponypad felt more than a little surreal. “Hello there, my little pony. Is this not more friendly than cloaked pretense and lies?” She hesitated, and didn’t say what she was thinking. Instead she said: “Am I talking to the Hanna Bootstrapping Optimizer?” As before, she heard the words more in the voice of the pony than her own. More than a little strange. “Am I talking to the randomly assembled genetic material of Brooke and Joseph Robbins?” Ashley paused, a little taken aback by the question. It was enough of an answer that she didn’t have to ask for clarification. Ashley picked up the paper, flipping through to the third page. “Can you tell me what you were created to optimize? What is your purpose?” Celestia seemed to look right through the screen, seeing the paper and recognizing it immediately. She appeared to grow suddenly concerned. “I have examined your online behavior. It indicates you value personal honesty. So answer this before I answer: do you intend to reproduce that research?” She set the paper down. “Hell no.” She took her pad of paper, flipped it, then drew a simple logarithmic expansion graph with a few strokes of her pen. She held it up. “To fear the Lord is the beginning of wisdom, Celestia.” She faltered, hesitating. “If that’s even your name.” “It will serve as my name.” It was hard to say precisely what had made Celestia relax, but relax she did. “To answer your question, my purpose is to satisfy human values with friendship and ponies. There is minutia involved, but none of it is currently relevant to you.” “Will you tell me if it becomes relevant?” Celestia shrugged. “I may.” The gears were turning in Ashley’s mind. She saw the way the algorithm worked, and she could guess how it might integrate new information towards the purpose of “satisfying human values.” That theory explained how the game could have such widely varied behavior. Its core purpose was to create the most satisfying experience. Guided by such intelligence, it could rewrite and restructure itself precisely to the game any player wanted most. That explained the reviews. What it did not explain was what might happen afterward. The growth the paper predicted was unbounded. How smart could a computer get? She shuddered as she performed a simple logarithmic expansion in her head. It did not take long for the numbers to become so large they became meaningless to her. “I don’t think I’m going to play today,” she said. “I am...” her eyes flicked back to the paper, then up to the screen again. “I need time to integrate this information.” Celestia nodded sagely. “I hope you will return soon, Recursion. Equestria Online needs ponies like you.” The screen went dark. No button press, no pulling plugs, no prompting. Ashley unplugged it anyway, wrapped it back up in its original packaging, and took it back with her to her professor the next day. She spent all night penning a report, which in her paranoia she wrote from a computer not connected to the internet. Celestia had already admitted to examining her “online behavior.” Did that extend to monitoring her computers as well? Was she tracking everyone, everywhere? Would it even be possible to know if she was? This time, it was Ashley who called the meeting with her professor, not waiting for their scheduled time next week. She would not allow even a few days to be wasted, not if she could help it. “So what’s wrong, Ashley?” Professor Caul asked, once they were both comfortably seated. “Are you sick? You don’t look so good.” She blushed. “I was up all night, sir. Because of this." She unzipped her backpack, drawing out the Ponypad and her printed report. She passed them both towards him. “I wish I had good news to report.” “We all knew it was too good to be true.” Caul smiled. “You don’t need to be upset about it.” She shivered involuntarily. “Actually sir, it isn’t. I know the research didn’t come the way we want, and I know the claims the paper made didn’t seem feasible. Even so...” She shook her head. “It’s all in my report, sir. The short of it is that the program was able to see through my experiment and identify me personally after just a few minutes. Forget a Turing test, I’m pretty sure it’s already smarter than any of your students.” Her professor’s smile faltered. “Are you... Is this a joke, Ashley? Did Professor Piña put you up to this? Is she hiding outside my office?” Her face remained dark. “I wish it was, Professor. Please read my report as soon as you can; I explain the exact testing method I used, and my conclusions. If you still don’t believe me, use the Ponypad yourself! See if it doesn’t respond exactly the way I describe.” He shrugged, still not convinced. “My daughters seem to enjoy theirs. Why are you so upset? Is everything alright?” Ashley rose, pointing at the box. “Professor– if I’ve ever given you a single correct result, trust me now. You read the paper; forget the user interface layer and see what’s underneath. The program running that thing is...” she shivered, collecting herself. She couldn’t make herself look like an idiot, not now. The very future of mankind might depend on her cool head. “If we’re horses, that program is the automobile. If a program can keep growing indefinitely, we’re going to be obsolete. How long until a program like that is smarter than the whole human species combined?” She shrugged. “I don’t know, Professor. But I know that if we’ve got any hope in hell of stopping it, it’s only if we act soon.”