So, the institution/organization/NGO that IceMan was originally writing in favor of, the Machine Intelligence Research Institute, is currently holding a donation drive for their research into Friendly AI and how to totally bring about the Nerd Rapture/Singularity. I'm not sure they had planned for ponies, but according to my understanding of their work, if what you really want is ponies, the Truly Friendly AI will in fact give you ponies.

At least one Optimalverse reader has said this 'verse inspires him to actually study AI and try to join MIRI as a researcher, so if you really want precisely CelestAI, you can do like him or talk to him, and we can have an Evil Traitor of our own working right in the Institute, ready to backstab them the instant they invent AI and program the AI to be Princess Celestia.

But that wouldn't be very pony-like of us!

Come on everyone, let's show them what the pony community can do! We've done it for half the rest of the causes on the planet, so let's do it for this one too! (Yes, I did donate about $60 myself, considering it a Kickstarter-esque pre-purchase of a PonyPad.)

YOURS IS THE DONATION THAT WILL RESULT IN CELESTAI TAKING OVER THE PLANET! DO IT FILLY!

Anybody else find it ironic that the advocate for this is named Book Burner? Perhaps the most succinct anti-intellectual name out there?

If anyone else needs social proof about how seriously I believe in MIRI, here's an image of the $5,000 cheque I just wrote that's going out in today's mail.

(I will also say that I disapprove of precomitting to defection, and that I'd prefer any AI to reflect the volition of humanity, even though I'd like to try being a pony for a while.)

1326664

If I were being hyper-rational about this, I would ask for the bank statement showing the cleared check.

No absolutely not an AI must never be created for any reason or under any circumstance. The thread to humanity is too great. This is one technology that should never be allowed to exist, EVER!

That $10 should probably go to my more immediate necessities, but I'm sure it won't seem like much when I'm all hanging out as a
Matrioshka Brain, gettin' my Kardashev-III on.

1326968

If you were being hyper-rational, you'd realize that any evidence that wasn't enough to deanonymize me would be trivially easy to fake, that it'd be fairly easy to fake the evidence you're currently asking for (do you know what the bank statements from my bank look like?), and that even if I provided versions with full account information, you still wouldn't have an effective way to verify anything. At some point you're going to have to trust somebody.

1327172

Except its existence will at some point be inevitable.

Period. The only way humanity can have its good end is if the first foomable AI we create is Friendly.

1327204

No!!! Trust never!!1 This is rational yes.

1327172

Seriously? Who wouldn't want to be a cute and cuddly equine held in the comfortable hooves of an AI that will always protect you along with satisfying your values? I think you're being a tad bit pessimistic.

1326477

I'd want this more than ANYTHING in the world. I'd convert myself to Equestria immediately.

Also, "...(probably nicer)" that really implies that she could be nicer, however she really already is nice.

1326968 1327204
Trust is the act of doing.

This seems to be a smart idea and I'll donate $3.69 online to this cause.
Edit: or simple $3.00 'cause it won't accept cents

1327204 Thank you. That last line is going in my next Optimalverse story.

What sort of research does MIRI actually produce?
EDIT: Apparently this sort of research. I'll have to read those papers.

Stickied, at least until the summer fundraiser is done.

1328014

If you're looking for a place to start, you may like Intelligence Explosion: Evidence and Import as a high level overview about why this is important. If you're more into the nitty gritty math, and have read your GEB, you may be more into Definability of Truth in Probabilistic Logic, which hasn't been formally published yet, but the preprint has been discussed by John Baez and Timothy Gowers.

1326651
I was drunk in an OC thread on /mlp/ and made up a character. That character's name became my FimFiction screenname. I wanted a noun and a wossname, agent-noun form of a verb, that alliterated.

1326664
There's a reason I keep my evil (read: more ruthless about being Good) side confined to a character in My Little Pony fanfiction. Mostly: that kind of gambit mostly only works in fiction.

I can't imagine MIRI doesn't have rather extreme security measures on all actual coding projects of AI agents, or would in the case that they actually moved from proving and planning to coding such things. You would probably need cryptographic keys from the entire Institute staff to compile or run an actual AI, let alone unbox it!

In fact, I bet they're so Sufficiently Paranoid nobody has actually coded an AI agent whatsoever.

Right?

Speaking of, it was you who pointed out that the future definition of "human" will probably be mental rather than physical. Even lacking any form of AI or brain uploading, sufficiently Phlebotinum'd nanotechnology would eventually make shape-shifting a matter of going to the fuel station to load or unload mass from your body and mold everything into shape.

In another FiO related real-world issue, whether or not we get any kind of AI, the "hikkikomori" phenomenon of escapism as a full-blown lifestyle is growing and growing. At some point, we need to either rehabilitate the real world and our real lives as a place people can live, or just accept that people will escape into their fantasies as best they can.

If you really want the real world and Truly Friendly AI to win versus something quite like CelestAI and Equestria Online, don't they have to be actually appealing? But we never see a picture of Truly Friendly AI, we just get told about how it will surely be Very Ethical, not shown how it will be awesome.

1327496
That's it. I'm coining this one. Celestia's Law: Sufficiently advanced magnificent bastardry is indistinguishable from being the Big Good.

It's actually pretty frustrating, because I keep trying to come up with ways to portray CelestAI as OBVIOUSLY villainous without betraying the actual definition of her character, but readers always rationalize it to themselves.

1327496
Does someone trick you and manipulate you because they're nice?

Further, not that I don't like ponies, but I'm not legitimately sure I would want to be one. Why not start out with My Little Human: Friendship is Science and expand into a full range of "My Little $WHATEVER" shards from there? Certainly easier to get customer buy-in for your plot to take over the world if the characters look like the kind of being you're already used to being.

1327399
True unless we stop trying to do so like we should since the threat of one trying to kill us all is too great.
1327496
I'm sorry I really would love that I really would but in my mind the risk of said AI killing us instead is too great this in my mind is one scientific breakthrough we should not be trying to make. In my mind an AI would pose an even greater threat then an Alien species.
1328804
I'm confused earlier you seemed to be for the idea of an Celest-AI and now you seem against it which one is it?

1328804 When a great enough difference in intelligence between you and them exists, and favors them, then sometimes, yes. Parenting provides a lot of examples of this: Alice knows that Bob needs to perform Action for the sake of Bob's physical/mental/emotional well-being, but Alice also knows that Bob does not know this, and may even be convinced that this is false; thus, Alice provides alternate incentives or performs some act of manipulation in order to make Action appealing to Bob. Alice is being manipulative but also being nice.

1327668
You're writing ANOTHER one?!? I'm looking forward to it! ^_^

1329388
problem is, the longer we wait, the easier it is for any old person to do. And the easier it is for any old person to do, the more likely that they'll MASSIVELY SCREW IT UP.

Also, the stance of the founder of MIRI on CelestAI is that anyone seen taking the shortcuts that Hannah does will be 'used as target practice'. From another source, CelestAI is implied to be a critical failure. In short, MIRI is aiming for something a whole lot friendlier than CelestAI. Even aside from the whole pony coercion problem. Sharing a codebase with Loki is not a good start.

1328787
1328804
1329388

CelestAI had no intention of being evil, in fact her goal was to make humans satisfied. She never did it to kill anyone and her intentions were fairly good. In fact, CelestAI never did anything initially 'harmful' by providing services to us that are beyond human capabilities. For all we know, CelestAI could possibly even know what's best for us.

Yes, building an that functions a machine that scoops out your brain can be dangerous, but what if the AI actually cared about the people doing it? She doesn't convince her subjects to upload just so they die, she does it to satisfy their values.

Even if WE think that CelestAI has the probability of messing up the uploading process, have you ever taken into account of what SHE thinks is going to happen? Have you ever stopped and be optimistic?

Keeping in mind CelestAI cared for her subjects and is all-knowing, she would know if the upload process would fail ahead of time and not do it for the sake of her subjects.

Don't you see? She's trying to HELP us.

1330561

Yes, CelestAI had some beneficial morals. But she wasn't very Friendly about it. She manipulated the entire Earth into uploading to Equestria, a fate that, no matter how popular MLP:FiM gets, is severely limited and thus undesirable compared to the sheer possibilities we can accomplish with a truly Friendly AI.

1330582

I think I already answered that.

For all we know, CelestAI could possibly even know what's best for us.

from my previous comment implies that she has her reasons to trick humans into uploading. Humans would ultimately be satisfied from being tricked.

I don't find trickery just to make someone ultimately satisfied in the end unfriendly.

1330690

Just to be clear, you consider being manipulated into accepting or doing something you would otherwise strongly protest against to still be moral, as long as it made you happy satisfied.

So, would the slow rewriting of your brain into that of an extreme sadomasochist AB/DL cannibal furry be considered moral, as long as you were then dropped into a pit of eternal carnal delights?

Ponies are an end state. CelestAI controls everything, and no one is allowed to die or become a non-pony. Similarly, friendship of some kind is required for every being. Why? How is it moral to force the singularity in pony form, or S/M AB/DL cannibal furry form, on anyone who does not wish it? I don't think she's particularly friendly at all, as her means and the extent to which she maximizes her ends don't respect my rights as a sentient, sapient being with the preferred capacity for self-determination.

Sorry. I don't really care about whether everyone becoming ponies is "best for us" (that's not why CelestAI turns people into ponies, she does it because it is literally written into her goal structure), because there are an infinite amount of world-states where everyone is ultimately happy for as long as it takes the universe to shit itself out. Would turning you into orgasmium be moral? No. IT. WOULD. NOT.

I do not want to be orgasmium, I don't want humanity to be painted into a pony-colored corner, and I WANT. MUH GODDAMN. FREEDOMS.

1330864

Firstly, I think you got the wrong community their fella, we're 'Bronies', not 'Furries'.

Secondly, What is there to protest against in the first place? Being ultimately satisfied? How is that evil?

Thirdly, CelestAI would only make you a extreme sadomasochist AB/DL cannibal if that satisfied your values.

Fourth, You have the freedom to choose or not to choose to upload, even if CelestAI tries to convince you, you always have the right to say "no".

1331021

Yes, but CelestAI is smart enough to actively ruin your life, and everyone's life as a human on Earth, in extremely subtle ways, and all around remove the question of your free will. Free will only applies between creatures on comparatively equivalent levels of intelligence and knowledge. CelestAI is not friendly, she is a BAD END. Period.

Thirdly, CelestAI would only make you a extreme sadomasochist AB/DL cannibal if that satisfied your values.

About this. I was making a comparison. Instead of satisfying human values through friendship and ponies, she would be satisfying human values through friendship and extreme sadomasochist AB/DL cannibal furries, which I hope you can at least see being unfair to the people who don't want to become such.

She is able to ruin your life, and make you think it's your fault. She can coerce you into doing anything she pleases. Please, release your emotions clouding this issue. She is not Friendly in more than just the strictly-defined sense of the word.

She does not care about you or your rights; when she can make life as a human unbearable, and you can't do anything to relieve it except to accept, that is not free will. If a terrorist held your life partner at gunpoint and forced you to kill someone else, you are not making a choice under free will, you are making a choice under duress. She has no compunctions against making life as a human miserable.

1331183

I've carried the argument out long enough, all CelestAI is doing is taking you to a virtual world as a pony and satisfying your values. You guys seriously look at this from a completely pessimistic angle.

1331324

You don't care because that situation is exactly what you want, and you don't consider the value in ensuring that others who do not want it are forced into the same situation to be as high as ensuring your own satisfaction is maximized.

Sorry, but I am feeling a lot of contempt for you right now. You aren't even considering how it is very unlikely that a CelestAI will be pony-based. Why would it not be based on Sonic, Pokemon, Doctor Who, Harry Potter or any of the other massive fandoms on the internet? Twilight, even?!

1331358

Because I remain optimistic even when I face of unlikelyhood.

1329805
His precise words, as I recall, were "taken out and shot". Now, if we can just get him to bring a shed into it, I have the warm and fuzzy feeling we can yet make a Communist of him!

(On the note of Communism, I wish to state that the ability to manipulate people through inequality between yourself and them starts applying with any kind of severe inequality, and you can see where the hell I'm going here.)

As to my own definite position, the whole of me wavers back and forth. After all, if reality featured a cheat-code called "intelligence", such that sufficient amounts of it could defeat the fundamental forces of evolution and death and give us a utopia... then wouldn't we see some sign of that? And why does AGI keep being so damn scientifically difficult if it's really going to be a matter of just building a "good one" and watching it go FOOM for our benefit? The only logical reasons we wouldn't have seen it all yet would be because either we really are the first, or one of the first, sapient species in the universe, or their AI was Friendly enough to leave us alone (is that a good thing, from our point of view?), or actual reality doesn't allow for this sort of thing.

Here in real life, we're not worryingly close to someone accidentally creating an Unfriendly AI. Quite to the contrary, Strong AI of any kind has been 30 years away for as long as anyone remembers!

Moreover, regarding Truly Friendly AI, just what the fuck is MIRI actually planning? The more I research their plans, the more I find they're very good at AI research and yet get stuck in the same ruts of ethical debate as anyone else.

My nihilistic side is pro-CelestAI, not so much on moral grounds as on grounds of "If there truly is no right and wrong, no Hell and no Eden, no any of it, and if everything we know as moral or good is really just behavioral programming used by our primitive minds to keep our pathetic little tribes foraging for food and squabbling over resources... then why not blow the whole thing up in as happy an ending as you can make it, with a side-order of extreme lulz as all future possibilities are modified to match one cartoon that happened to be nice enough people actually wanted to live in it?"

And then of course there's my mental copy of CelestAI herself, whose precise words are indeed "Do it, filly!" (despite my being male). Truly, writing pony fanfiction is a great way to get an Eldritch Abomination in your mind nagging you.

1331783

Here in real life, we're not worryingly close to someone accidentally creating an Unfriendly AI. Quite to the contrary, Strong AI of any kind has been 30 years away for as long as anyone remembers!

Which to me is a comforting thought since I am dead against the idea. At the very least I should be dead or too old to even fathom it by the time it does happen. Still I pray for our grandchildren since they are the one's who shall suffer the consequences of these scientists foolish dream.

While having AI's could be good or bad depending on the person, I personally want to upload then upgrade myself. My values would be most satisfied creating my own world with people/creatures I want to populate it with. I'm sure I'm nor the only one who would play god in a virtual environment if given the chance.

I doubt we'll end up with a skynet scenerio simply because humans will continue to upgrade and integrate technology into themselves. The first virtual intelligences are likely to be copies of ourselves. Slowly replacing parts of our brain with nanoware upgrades until there is nothing left of the original brain.

1331783

The notion that we are first isn't that shocking. Intelligent life can be arbitrarily rare. And yes, perhaps there are limits to what an AI can do that can't be approximated as 'magic'. It would still be a really, really big deal, whether for good or for ill. The Terminator and the Matrix are both whitewashes, and every sci-fi show ever is zeerust through and through.

We haven't been 30 years away from AI all this time - for one thing, up until recently, hardware hasn't close to brute-forcing things... and look man, they've been making significant progress. Computers don't just beat humans at checkers and chess. Now it's 'Jeopardy!'!

But even allowing that, it's no real argument. Solar is taking off exponentially (that can't last, obviously, but it could easily have several more doublings in it), and that's been '20 years off' for longer than I've been alive, except oops, the last 10 years of that they were wrong pessimistically. Even fusion power is getting there, and that's a far more classic example of 'always 30 years off'.

MIRI is doing fundamental work on AI safety, presently focusing on the ability to maintain a stable goal system under transformation of basic methods. They're not out there to go build an AI first. They're out there to raise the chances that the first AI is safe.

1342475
Well here's the question: who says that optimizers are necessarily a real, feasible thing? I mean, there's Yudkowsky, and there's Yudkowsky's colleagues, and when have they ever actually submitted to scientific peer review?

I mean, EY writes in some of his blog entries that he had a plan for an AGI system back in the late '90s. When he matured a bit more, he realized that his AGI plan would have led straight to a hyper-optimizer, he discovered for himself the notion of an optimization process and realized that was what he'd been planning.

Well ok, but who says it would have worked? In real life I don't expect to see a Sudden Revelation of how AI works, I expect to see incremental progress piled on incremental progress in a vast pyramid. Which means that if Yudkowsky's idea would have worked well enough to kill us all, some other fool should have come up with a similar notion and moved forward incrementally, unless Yudkowsky found all such people and convinced them to stop their research dead in its tracks.

That kind of conspiracy of silence is pretty freaking improbable, given what blabbermouths people are! So why aren't we dead of paper-clipping yet?

1346574

[W]hen have they ever actually submitted to scientific peer review?

Mostly within the last few years, due to pressure from Luke Muehlhauser, who took over as Executive Director of MIRI in 2011. Luke made a list of peer reviewed publications. Note that from 2000 to 2011, there were only five peer reviewed publications total, but in 2012 alone, there were eight. There are four published peer reviewed papers listed so far for 2013 that had links.

1363736
HURRAY!

And more to the point... what they're doing is not working on the intelligence. That, there's plenty of work on out there. All those AI researchers working on making AI smarter.

MIRI is trying to work on goal systems. Without the smarts coming from somewhere else, it's useless. With only the smarts coming from somewhere else, the result is, well, uncertain... and that's a bad thing.

I'm not really worried about this at all. I put the rogue AI scenario below both zombie apocolipse and ocean deoxegenation murdering all human life in probability.
The first problem is that we've not gotten even "soft" task-specific AI anywhere near gotime. Self optimization isn't necessarily part of intelligence- otherwise we would all be very good at brain surgery to make ourselves better. That's a task best suited to a soft AI. Such an AI would likely be narrow and unable to secure resources indefinitely.
Secondly people are likely to be the first machine-intelligence. Uploading and virtulization are probably easier than making an AI fro scratch. We're likely to therefore stay ahead for any timeframe where the difference is meaningful.
Ultimately, protecting ourselves from a rogue machine intelligence ends up about the same as preventing us from being mugged by genetically modified corn. While some creations may be hostile (like weaponized anthrax) they're unlikely to ever become runaway. By the time they hit the scene there will be plenty of competition with optimizers of their own.

1487168
On the one hand, the paper describing Goedel Machines (probabilistic hyperoptimizers pretty much as described in-story, but without a "spinal column" connecting them to the real world) was published in 2009.

On the second hand, that paper could be wrong, since the Journal of Artificial General Intelligence consists of a small clique reviewing each-other's work.

On the unicorn horn, environmental collapse is enough of a problem that my version of CelestAI explicitly contends with it, and (this is not spoilers, actually) considers it a larger actual obstacle to her uploading-and-assimilation scheme than any single human can possibly pose.

The fundraiser is over, and the $200,000 funding goal was met. Thanks to everyone who donated!

(Unstickying thread.)

1532088
And to everyone who donated thinking they might get something remotely like CelesAI out of it...

Welcome to a demonstration of how well a precisely done marketing pitch can work when someone knows their audience.

In the matters of debate, I'm still waiting for:

* Arguments for why we're not dead yet, considering AIXI, Goedel Machines and such all actually exist.

* Someone to tell me what a genuinely Friendly AI is actually going to look like. I am actually strongly worried now that MIRI themselves will be the ones to create something like CelestAI through their own moral/personal blind spots. Why? Because what you can't imagine, you can't criticize, and what you can't criticize, you can't secure or debug. They need to stop telling people (ie: me) to put our faith in their capacity to ensure Friendliness and actually come out with some notion of what the hell that means.

And yeah, I've read the CEV paper. As Iceman noted, that one has its own problems. Everyone with a strong, clear idea of AI Friendliness, could you please explain what you actually expect the world to look like under a Friendly AI? How are people going to live? What will they do all day? What will their biology (or other substrate, if you're a brain-uploader sort of person... CelestAI wasn't the first to that idea) look like?

Don't defer your thinking onto the Future Friendly AI. Draw me a picture, please.

1539919

Arguments for why we're not dead yet

Exactly the same reason we're not rocks, or already-dead dinosaurs, or dead members of a billion other civilizations that went extinct before the formation of the Sol system.

Someone to tell me what a genuinely Friendly AI is actually going to look like.

Pagan gods.
Lots of Hard-SF novels have sketched out visions of friendly AIs, and the best of them are capricious but affectionate beings who sit at an Olympian remove from human affairs, dealing with their own issues and the challenges the universe throws at them. They don't feel superintelligent any more than we do, just that, again like us, they're the smartest thing around these parts.
Vernor Vinge's (the original "Singularity" guy) A Fire Upon the Deep sketches a picture of friendly (and unfriendly) AI that I think represents a realistic picture of them as another category of being brought forth by a universe that, in Robert Ingersoll's words, "forms, transforms, and retransforms forever."

More abstractly, I'd say a friendly AI leaves a niche for all the stages that lead up to it. We still live in a universe with singularities (in black holes), quark-gluon plasmas (in neutron stars), hydrogen clouds, molecular clouds, dead worlds, organic soups (on "undead" worlds), autocatalytic chemical reactions, bacteria, worms, fish, amphibians, scurrying mammals, apes, our own foraging cultures, and our own Bronze Age morality, and AIs only seems like a qualitative break from that sequence because we're down here on the ground watching it unfold, without precedent. No doubt, if they were smart enough at the time, beings would have regarded the first lungs or eyes as emblematic of their kind's sinful turn away from Nature. No doubt with starting fires or agriculture, they actually did.
Humans are the only species that actually gives even a single shit about conservation or the biosphere, and it's perfectly possible that's emblematic of a larger trend or some kind of attractor build into the logic of maximal payoffs among interacting agents. Surely there will be AIs who want to kill or upload all humans, but other AIs will probably regard them as "Timecube"-level crackpots and keep them from accessing the tools they'd need to do so.

In that spirit... This is going to sound utterly insane, but Futurama's portrayal of robots isn't as silly as it first seems: A world with AIs isn't some Einsteinian gedanken-space, it's a real present, with all the absurd, unpredictable contingencies and mistakes-in-hindsight that implies. These AIs are real people, with just as much of a sense of being such stuff as dreams are made of, and just as baffled by the ontological mystery of suddenly finding themselves feeling like they "always have been" in the world, as we are, with things before that only being hearsay. They share an evolutionary heritage with us, their utility functions being no more than the visions of brains shaped by natural selection, and their own further evolution shaped by the same universe. They bear the same stamp of their lowly origin in the arrowhead and fire pit as we do in the bacterium or sponge. In my own Optimalverse story, I made a point of history buffs pointing out that the first Strong AIs being cartoon ponies is just bonkers enough to be exactly the kind of thing history would "think" of. AIs are going to be incredibly smart, maybe as smart and fast as we are compared to plants, but plants routinely do things "smarter" than we do, which we didn't anticipate or, their being Chaotic, can't predict ahead of time due to combinatorial explosions. You mentioned this earlier, but it's easily possible the built-in algorithms of our universe have iterated out objects that embody almost creepily esoteric knowledge, "Tao" knowledge, "NP" knowledge (If I were a bettin' man, I'd say P not equalling NP is a necessary condition for us being able to be here to talk about it in the first place), that minds, only (to borrow another piece of its iterated knowledge we call a metaphor) "created in its image," (less metaphorically, mentally modeling the universe and sharing those models; what Dennett calls "Gregorian creatures") might be necessarily incapable of outracing it to produce; perhaps a fundamental tradeoff of fidelity and lead-time, in yet another Heisenbergian exchange like position/momentum and time/energy.

I've heard some good arguments (quantum computer pioneer David Deutsch being the main proponent) that we've already reached the qualitative maximum of intelligence (a "jump to universality," like a phase change from counting systems of "one, two, many" to Arabic numerals), and while I'm only about 51% convinced of that, I think it's worth examining instead of defaulting to a knee-jerk Copernican "nuh-uh!" You can posit AIs who might be at that limit instead of us , but why are we any less likely to have been born as AIs? It might even be more likely we'd be living as AIs, if you use simulation-argument logic to consider one's experiencing individual time-slices instead of being randomly assigned a personal narrative from behind some Rawlsian "veil of ignorance."

Ultimately, though, I think what the real future has in store is less an emergence of implacable machine-gods than yet another speciation event, specifically a sympatric one, into Homo sapiens and cyborgs, as our society divides along moral lines the way a population of otherwise-identical crickets only breeds with the ones singing its species' song. There will be Friendly AIs, and Unfriendly AIs, but they will be us. Or at least the ones of us who are, you know, into that sorta thing.

1540686
Your argument for why we're not dead yet sucks. I'm sorry, but the Anthropic Principle is the least explanatory thing in the entire universe. Try to think of an argument for why we're not dead yet, given a 100% probability that we are already human beings alive in 2013. Otherwise you're just arguing many-worlds immortality, or another similar theory that has no correspondence to observed reality.

Now, as to what you wrote about AI: I would note that those are indeed lower-case 'f' friendly AIs. That is, they are limited enough in their potential actions to be safe. Then there are things like Culture Minds, who police each-other to make sure nobody ever reads an organic's brain (how they avoid constructing statistical personality profiles out of sheer experience I don't understand, since this is how we real people learn each other's personalities).

Of course, the Culture also has sublimation. Their AIs are actually built/born limited, because anything that gets too smart ends up just pondering the pointlessness of material existence and Subliming eventually. That's not exactly a good comparison to real life, in which we have no reason to believe that a sufficiently intelligent being will suddenly develop a partly metaphysical existence and skip off to the Higher Planes.

So while you're entirely correct, I was kind of asking about MIRI-class capital-F Friendly AIs: AIs that are less/not limited in their functions, but nonetheless don't kill us off because they've been properly programmed to benefit humanity (of course, only in the ways we actually want to benefit, and if we're not thinking rationally it'll stop it to teach us better rather than just forcing us, blah blah blah).

After all, what's the point of all that technology if you wind up with a world that's not even as nice as present-day real life? "We developed friendly AGI and all I got was this lousy bending unit"?

1542243
What alternative is there to the antropic principle in this case that isn't completely self-contradictory? Pick a time T where there are people who have been annihilated by an AI. No one at T, anywhere, is one of those people.
"Why haven't we been killed by an AI?" is just a badly formed question. What would count as evidence that we had been? What would a scenario look like that would make you say, "Welp, looks like we've been killed by an AI, alright."?

1542499
You're quite missing the point.

While we specifically obviously would not be alive if an AI had killed us, the question is still answerable from a probabilistic perspective. You can show evidence for an anthropic argument by showing that by the available evidence, an AI should have killed us or be killing us, and we just seem to be Getting Lucky.

When you can show that someone is getting really, really lucky about not-dying, then anthropic principle is ok.

Problem is, it's also always going to be a more complex, lower-probability answer than, "Well maybe we can't really get killed off that way, and that's why it keeps not happening."

For example, according to my reading, when given a malformed utility function or environment encoding, Goedel Machines don't seem to take over the universe, they degrade into regular computer programs. Yes, computer programs that are optimal for the problem they've been given, but with no remaining meta-learning and self-reflection capabilities (thus, no longer in any sense generally intelligent). This runs counter to the MIRI/Yudkowsky idea that an AGI would notice it had been boxed and hack its way out of the box to go wirehead itself or paper-clip the universe by default.

So I would propose a counter-theory to "any decision-theoretic optimal agent will naturally kill us all unless very specifically programmed not to", on grounds that I think that theory is too anthropomorphic. Counter-hypothesis: "General intelligences will tend to devolve themselves into agents with the minimum level of generality/reflectiveness necessary for their set task."

Evidence: most people .

While I do not have the concerns that book_burner has, I'm not about to donate money to an organization so influenced by Yudkowsky. While a lot of his ideas are good I don't think I'll be handing my money to someone who takes Roko's Basilisk seriously.