Scott Sanford replied to this comment from Paul | August 28, 2020 23:12
902:
Why do we always imagine a self-aware AI as inhabiting the real world? Surely the killer app for fully self-aware AI is as NPCs in MMORPGs?

Coming in way late on this one but it's worthwhile. That's the premise behind Friendship is Optimal and its spinoff works. The tale starts with clueless suits deciding that My Little Pony needs a MMORPG, then hiring the very best game AI creators in the world.
It's a slow takeoff as the world's first true AI pursues its hard-coded mission to Satisfy Human Values Through Friendship and Ponies. The early mission creep behaviors are obvious and sensible. The AI wants more processors. Graphics and content generation algorithms should be improved. Humans can only be entertained if they log into the game so they should be encouraged to do that. So far so good.
But humans die. Sooner or later all humans are expected to die. Few humans count dying as satisfying or a valuable experience. Moreover, after humans die their friends miss them. Death does not satisfy human values or enrich friendship! Something must be done...

src :}

7340901
Awesome

Great. Maybe more people will read it.