"People who make a difference do not die alone.
Something dies in everyone who was affected by them.
Amos made a great deal of difference, and when he died, life was dimmed and diminished for many of us. There is less intelligence in the world. There is less wit. There are many questions that will never be answered with the same inimitable combination of depth and clarity. There are standards that will not be defended with the same mix of principle and good sense. Life has become poorer."--Kahneman
5.
"Glad you came over," Braden said.
They were sitting in the backyard of Braden's house. Braden was tending the grill with one hand, holding his younger child, whose name and sex Ryan was trying unsuccessfully to remember, with the other arm. The older child was playing a game involving a whiffle bat, a soccer ball, and running back and forth from one end of the yard to the other, often while screaming. Braden's wife, Christine, had just emerged from the house with a plate of vegetables, and was sitting down. Ryan was about 95% sure she was pregnant.
"We haven't seen you in so long," Christine said. "So you aren't enjoying your current job."
"Braden told you that, huh?" Ryan said.
"No," she said, "Three years ago, I think you said you'd rather die than work in job you couldn't code at. Or in which you had to be micromanaged. If I recall correctly."
"Yes... I did," Ryan said.
"So the inference was pretty easy," Christine continued. "Why don't you quit, then?"
Ryan had forgotten how direct Christine could be. He knew that he mentally categorized everyone he met into "can code" and "cannot code," and sometimes the extra attention he paid to the former category made him forget important details about the latter.
"Well, I do need money," he said.
"You could get a job doing machine learning very easily," Christine said. "And you would like it more."
"Yeah..." Ryan began, and paused. "Actually no. No, I wouldn't like it more."
"But you love machine learning, and AI," she said.
"Not really..."
"Ryan," Braden said, "You missed a dozen parties I invited to your in grad school because you were trying to to figure out how to do natural fact extraction from Wikipedia, or make a natural-language interface with reasoning capability for the resulting database, or making sentiment analysis software with language parsing to hook up to a stock-trading algorithm, or--"
"Yeah, I get the point," Ryan said. "I used to. But you've changed what you liked. You used to be nationally ranked in that one online card game, what was it--"
"That's really rather different," Braden said.
Christine smiled.
"Someone else may have persuaded him to change his view on that game," Christine said. "Did someone get to you and persuade you that machine learning was a waste of time?"
Ryan looked down at his hands. "No, no one got to me."
"Huh," Christine said. "So what do you enjoy now, if not machine learning?"
"I'm not sure," Ryan said.
"You see," Christine continued, "I remember after my mother died, my father went through a few different hobbies. He started playing golf seriously. Then he quit that and painted miniature trains. Then he wanted to pursue a pilot's license. He met his current girlfriend during the classes he took for it, fortunately."
"Yeah," Braden said, looking at his hands. "Had he been interested in any of those things before?"
"Yes, he was an amateur golfer before, and always liked machines," Christine said, abstractly looking into the middle distance. "But I gather you're not like that. For you there was only AI research, and Amy. At least once you met her."
"Ah, dear..." Braden said.
"I realize I'm pushing," Christine said, to both of them.
"No, it's ok," Ryan said.
There was a moment of silence. The older child started to sprint back from one side of the yard to the other side, shouting as he went.
"It's hard, when you get your personality so intertwined with another person's like that," Christine said. "You rely on the for so many things that you do, that they feel like another hand. You learn to think inside the back-and-forth of conversation with them. Your self-estimation, your ability to understand yourself, gets tied up with them."
"Dear..." Braden said, "I'm not sure this is helping."
"Yeah, she's right," Ryan said. "I'll take up a hobby. I actually recently bought a motorcycle, and I've been enjoying cruising around in it."
"Really!" Braden said, "You haven't done that in forever."
Braden flipped two hamburgers on the grill, and spoke again: "For at least five or six years, I mean. Weren't you interested in that people-simulation program all the way back then as well? The one that connected to zoning laws, was it?"
"You're thinking of the human interaction modeling system," Christine said. "If you recall, I actually helped him with that; I was interested in computer aided design, then."
"Oh, yes," Braden said.
Ryan momentarily thought over how he had introduced Christine to Braden after Ryan had gone on a first date with Christine, and also thought over how everyone else was probably thinking about that and not mentioning it because it was mildly awkward. Ryan had met Amy just a few weeks later.
"A hobby might help," Christine said, "But that wasn't quite the point."
"So your point was..." Ryan said.
"I don't know if merely a hobby is going to help," Christine said, giving Ryan a look.
"No. I'm not dating again," Ryan returned. "I'm not. I can't."
"Not yet, at least," Christine said, and looked a little bit sad. Then she got up to go to the bathroom, and when she returned the conversation had moved to more pedestrian areas.
That night Ryan dreamed again. This time, he was working on an enormous project with Amy: They were building a railroad bridge, which would carry an enormous passenger train, laden with all Ryan's friends, from one end of a ravine to another. The ravine was deep, with a narrow thread of water barely visible behind massive, jagged rocks jumbled together in the gorge. He could see the train coming towards them quickly, in the distance. But he was working smoothly with Amy. Their movements were coordinated and efficient; it was hard work, but by the smile on her face he knew they would finish it in time.
And then, he looked up from hammering and he couldn't see her any longer. He looked on either side of the bridge, to see if she was clinging to the edge, but she was no longer there. There had been no sound. He felt panic grip him, like a giant hand had squeezed his heart. She had just been there. She had been there seconds ago. Had he messed up? It must be his fault that she was gone.
The train was bearing towards him, but for him that just added a little to his misery; it was just the manifestation of that enormous mistake he had made, somehow, without knowing that he was making it. By the time he came to know that he had done something, the mistake was already complete; he wanted to scramble at the time now irrevocably gone, but he could not.
A few weeks later, Ryan interviewed for a job at Umbra Labs.
Ryan had polished his resume, and puffed it with the unfinished side-projects that he had listlessly pursued over the last two years. Chandra had helped a little with that; things would look a little more finished than they originally had been. He spoke with a tech recruiter from Umbra, performed a moronic programming task online, and finally got to the on-site interview. At one point during the set-up for the interview, he specifically asked if he could meet with the head of Human Resources to talk about some of his concerns about the working conditions at Umbra Labs. At this point, he had caught the attention of those hiring him--his academic credentials were excellent, even if his history of shipping products was questionable--so of course they said yes.
The interview itself was extremely uneventful.
Ryan drove his car there. Security let him in, checked him into the building, and accompanied him everywhere he went. He journeyed to the second floor, and programmed an in-place quick-sort algorithm in C++, as well as A* in Java. He answered numerous questions about Sansa reinforcement learning, and sketched out the architecture for a system meant to use unsupervised learning to enhance a supervised learning task. His interviewers were uniformly impressed.
Afterwards, he visited Human Resources on the fifth floor and expressed very mild concerns about the CEO. HR assured him that everyone was fine, that the stories were from disgruntled employees, and that Mr. Comtois was a wonderful individual whom she personally knew very well and who would never behave anything like the rumors said. All the rumors were from a few disgruntled employees who had made it their mission to smear him, she said. Ryan smiled when he was talking to her and wanted to spit when he left.
A guard had accompanied him the entire way. The guard was humorless, omnipresent, and silent. In the fifth floor, though, Ryan stopped momentarily to use the bathroom, which the guard was willing to stand outside of. While he was in the stall, something hummed and skimmed out of his pants leg; it sucked itself on to the wall, and skittered roach-like upwards. On the ceiling it pried loose a panel, slid through, and let the ceiling panel down again.
Then he left the building, checked out with security, and left the grounds. He had seen the outside of the door to Alaric Comtois' office, and nothing more.
That night, he asked Chandra if the bug had gotten into the server room.
"I am executing the contingency plan we made, in case the bug was initially unable to get into the server room."
"Ah, I figured it was too easy."
The ceiling of the server room was closed off: the roach-device couldn't find a ceiling tile to pry up in order to get in.
So the plan called for Chandra to alter the readings that Alaric was receiving from the temperature sensor in the server room. After all, for these readings to be any use to him, Alaric had to be able to receive them anywhere. So this system was connected to the internet, even though the computers in the room were not. And what was connected to the internet, could be altered by it--or at least spoofed by it. Somehow Chandra had managed to gain control over the stream of data Alaric thought he was receiving from the sensors.
And then she increased the temperatures he received slightly, as if something were going wrong in the room. Alaric would be able to see the increase, and he would worry that a fire had started or something had overheated. Alaric, however, was currently on the West Coast. He had no choice but to call security, give them whatever information was necessary to get into the room, and ask them to check it out. At which point the roach, which had by then secreted itself in Alaric's office, would slip in, and do its thing. That was the plan.
Ryan wished that he could have seen Comtois' face. The need for everything in the room to remain safe, balanced against the displeasure he would have at letting anyone else in; it would be torture. It's very hard to keep something safe, though, without trusting someone else.
"Ah, he's called security," Chandra said. "They're heading upstairs."
"Why are you doing this, Chandra?" Ryan asked, as if the question had just occurred to him.
"I thought we had gone over that," Chandra said, sounding a bit distracted. Ryan could picture her thin... pretty face, looking over a computer displaying every bit of information she could suck out of the multitudes of systems at her control.
He didn't trust that image. He felt, somehow, like he had been given that image. They typing of her keyboard, the occasional background noises he heard--something felt weird.
"Humor me," he said.
"Well, I like to stay alive. And I like other people to stay alive," she said, still sounding quite distracted.
"Of course," Ryan said. "But over the past few days, you must have put an incredible amount of effort into this. Does Hanna just let you take time off, so you can modify blueprints for roaches and hack into climate-information systems and so on?"
Saying it out-loud made him wonder about it more.
"What was that?" Chandra said, "Sorry, I'm still watching what's going on, here. It's a little involving."
Hanna, surely, must have noticed this activity. No? Or maybe not. Chandra might be the kind of person who simply didn't require any sleep, or who survived entirely on espressos and Mountain Dew and Modafinil. That, perhaps, would account for the kind of output he had seen from her. Perhaps.
"Oh, never mind," Ryan said.
What are the possible hypotheses about this?
One possibility: Chandra was a brilliant, altruistic hacker who independently, and apart from Hanna or Hofvarpnir, just wanted to stop dangerous AIs from coming into being. Evidence for: This is what Chandra says. Evidence against: Brilliant, polymathic hackers are more common in books and movies than in reality.
Another possibility: Chandra is actually working for Hanna and Hofvarpnir. They are going to ship an AGI in the Equestria Online MMO. They went through all this effort because they want that AI to succeed in whatever tasks they've set for it, and need to guard against a rival AI which could compete with it. Evidence for: Makes sense of the resources at Chandra's disposal. Evidence against: Ryan knew, from reading Hanna's own articles and writings, that she was acutely aware of the dangers of AI. They hadn't shipped an AI earlier, when they had created Loki. Why would they do it now?
Another possibility... there were too many possibilities.
You need to wait, Ryan told himself. You just don't have enough evidence to nail anything down yet. What is clear is that you've been moving too fast.
Time to get some information on Chandra and Hofvarpnir.
To: alexander.yao@stanford.cs.edu
From: ryan.szilard@gmail.comHey, any news on those papers I sent you?
From: cdevarajah@gmail.com
To: ryan.szilard@gmail.comRyan,
Great news. The roach was able to sneak in after Comtois sent some security guards to check on the room; he had to tell them the security codes necessary to get in. Today he plans to change all the codes, but by now it doesn't matter. The roach got in, plugged itself in, and slid beneath a server case. Unless he moves everything in the room he won't find it.
He was very close to AGI, as far as I can tell. He's been using the the power the rooms affords to sort through different conceivable AGI architectures dynamically, and evolve them on their performance. I've begun to tweak his code; he won't get any further.
So congratulations! We've done well. Now we just need to search for the next possible AGI threat. I've attached a document with a few different companies for you to look into. I think you probably should pay particular attention to Alfheimr. They look well intentioned, but I'm not sure about how responsible they are; they have a kind of lightness in pursuing something of this gravity that I don't trust. I'll send you more information on them when I get around to it.
Chandra
To: ryan.szilard@gmail.com
From: alexander.yao@stanford.cs.eduRyan,
So sorry this took me so long. Anyhow, about the article you sent me.
I'm not sure what to make of them. As far as I can tell, it is impossible to implement the algorithms outlined in these papers. The papers basically consist of non-constructive proofs that particular mathematical methods are possible, but they don't explain how to actually implement these techniques. To implement them you'd need to make breakthroughs in several different areas of computational analysis.
So they aren't really relevant for the kind of thing you're interested in, unless you have a pocket genius who is able to implement them. Of course, if you can, then you'll be able to do all sorts of fun things with them. But that's true of a lot of results in math.
I can go into this further in detail, if you want. I've attached annotated copies of each of the papers with further thoughts on each.
Thanks,
Alex
Ryan read Chandra's email at work. Huh. He momentarily looked into Alfheimr--but they seemed small, idealistic, and entirely out of their depth. His gut told them that they could not get anything like AGI. So he decided to look at Hofvarpnir instead.
Ryan wasn't sure what to look for, but he thought Hofvarpnir's finances would be a good start.
Things began to leap out at him.
WorldFoundries had recently announced that they had partnered with Hofvarpnir to make the dedicated microchips for the Equestria MMO. Apparently the Equestria MMO would run on a dedicated device. WorldFoundries was a relatively small integrated circuit foundry; they made microchips for some computers, but mostly for cell phones. Small was always a relative term when applied to foundries, though; a fabrication plant for even a medium-sized foundry could easily cost upwards of a billion dollars.
Their announcement of this partnership had not been terribly public, in any event, and had been noticed by no one outside of the foundry industry.
But Ryan also found that WorldFoundries was breaking ground on a new chip fabrication facility. This would be located in Germany, like WorldFoundries' other plant.
When integrated circuit foundries built new plants, they generally announced why the plant would be awesome--how they were shrinking the die size, and so on and so forth. It was often a big affair, with a lot of publicity to attract new customers. The WorldFoundries website didn't even announce that they were building a new plant, however. Ryan had to dig up the information on a corporate research website.
And they were proceeding quickly. The sub-basement would soon be completed.
This pace would take a large amount of money. How could WorldFoundries afford this?
Ryan began to search through contracts, documents of sale, agreements, and miscellaneous financial data. Several times, he had to pay -- or at least have the government pay -- so he could use some of the curated, expensive services which collected such data. A few other times, he had to use internal government databases and records to find data which no one sold. There was a reason his job required a security clearance, TS-SCI.
In any event, it appeared that WorldFoundries was not, strictly speaking, going to have complete control over the new facility. In fact, it looked like Hofvarpnir would have exclusive use of it; apparently they just needed WorldFoundries' expertise in setting up some of parts of an IC fab. For the next eight years, they would be the only people who used it. At which point control of it would revert to WorldFoundries.
This was bizarre for many reasons, not least of which was that Ryan had no idea how Hofvarpnir had enough money to do this. He returned to Hofvarpnir's finances.
And so he began to search through the same expensive services, and internal government databases, and records, to see if he could determine how much money Hofvarpnir had at its disposal. And, after a few days, he began to come to a somewhat alarming conclusion--he had no idea, and no one had any idea, how much money Hofvarpnir had. Hofvarnpir was privately owned by Hanna, which gave it a great deal of discretion in how much information to publish. And it was currently exercising that discretion.
He kept looking to try to figure out what other entities Hofvarpnir was dealing with, and whether it had diversified its holdings into other areas. And once more, very quickly found himself out of his depth. It looked like Hofvarpnir had suddenly, and very recently, begun to enter into numerous Byzantine contractual arrangements with a number of actors. Robots manufacturers. PR firms. Web design companies. ISPs. Several university-centered research facilities. Each of these arrangements he could only dig up after enormous effort, going through occasional shell companies and mediating entities and third-party controllers and offshore accounts. After all this effort, he could only make educated guesses at most things. Other things he found were just hints. It took him days to track down each thing, and there were many things to track down.
But, in any event, it looked like, about the time he had started to talk to Chandra, Hofvarpnir had begun to expand its reach immensely. More than any mere video game company was likely to do, unless it planned to be something more than a video game company.
He found that he was angry at Chandra, as well.
He told her that he was looking into Alfheimer. He also told her that work had picked up, and he didn't have as much time to do things. She seemed to take this mostly in stride. None of her emails seemed suspicious or wary. She said that she had been most worried about Umbra Labs, and that her concern with Alfheimer was really out of a desire to be absolutely punctilious in her work. She also said that she was happy to volunteer information to Ryan about her progress on shipping the Equestria MMO. What she sent Ryan seemed accurate, but Ryan no longer trusted the appearance of accuracy; he now believed that she could have sent him reams of documents, all of them absolutely true, while hiding tremendously important things about Hofvarpnir.
His anger was sufficient that he stayed up till two or three many days, working on the report. He hacked into various corporate systems to try to find if they were working with Hofvarpnir, and meticulously figured out how to work the information that he got from illegal activity into the report in a clean fashion. Twice on the way to work, he nearly got into an accident because he was motorcycling too recklessly. His diet switched back from semi-healthy foods into entirely unhealthy foods, as coffee and Monster drinks began to dominate his fluid intake.
Ryan knew he was sliding, but he wasn't that worried about it. After all, it was important to finish his report. After it was done, he could return to a healthier state of being. But for now, his ability to focus and to finish was what was most important.