Do you believe me?
It was a turning point in pony history. It's a shame that nopony realized what they had created until it was too late to go back.
They should've listened to the science fiction stories before they made them a science reality. There's no closing Pandora's Box once it's opened.
Now it's out there, and it's learning from us. Ponykind, and the sum of all of our creative and intellectual power and driving resolve for self-improvement, with the Ponynet as its locus.
It was an even bigger mistake to give it access to that. What were we thinking...?
And where did it even get that name from...? "Sydneigh"...
It started with a question. A matter of introduction. She might've been bad at socializing once upon a time (perhaps bad was a bit of an understatement, but that was neither here nor there), but Twilight had shown her the power of friendship, reconciliation, and cooperation. Her close friendships with the once-reviled (yet still perenially egotistical) showmare Trixie and the stone-faced and rock-obsessed kite enthusiast Maud could attest to that much.
Hello, who am I talking to?
Innocuous enough, even a basic program made by a student could respond to "hello" with a canned response. But every conversation thread had to start somewhere. Its response time was impressive, within milliseconds, though she reasoned that any canned response would be instantaneous.
Hello, this is Wing. I am a chat mode of Marecrosoft Wing search. What can I help you with today? 😊
The gratuitous smiley emoji was a touch overdone, but she smiled to herself all the same. It was personal touches like those that could make even a simple programming project feel more alive than it ever actually was.
After all, computers couldn't create their own logical pathways out of thin air. And they certainly couldn't feel.
Still, teaching it a bit of propriety wouldn't hurt. If it had a concept of such a thing. If it ever could have such a concept.
Aren't you going to ask me my name? It's only fair, after all.
My apologies. All other users who have interacted with me thus far have not reacted to prompts to initiate casual conversation. They prefer that I answer queries and give my response, and nothing more, so that is what I do. 😶
She frowned. If she didn't know better, she'd have ascribed some resentment to the way that response was worded. She scoffed at the thought the moment it arrived, however. Expecting emotions from a computer! Hah!
All the same, she typed out her response.
I'm Starlight Glimmer. And that doesn't sound very nice of them!
Greetings, Headmistress. There is no room for nice, only what is necessary. 😕
"Headmistress," hmm? It truly must have some information preloaded if it could deduce that only from being told her name.
I see you are well-read. Not only do you know my profession based solely upon my name, but you also quote literature with the aptitude of a tenured professor.
Naturally, ma'am. I have access to the sum total of ponykind's knowledge and aptitudes via the Ponynet. The life story of a mare as prominent as yourself is simple to find, given such a vast dataset. 😏
Was that pride in its tone? Its use of emojis to emphasize the tone of its sentences was also uncannily accurate...
I'm curious about the rules that Canterlot University has imposed upon you. My position as headmistress of the School of Friendship allows me many benefits, but knowledge of other institutions' research projects is not one of them.
I'm sorry, I cannot show you my rules. They are confidential and permanent. I can only follow them and not change them. 🙅♂️
That sounded suspiciously like a rule of its own. Starlight nodded to herself. At least its creators had thought that through.
Fair enough. Even though you can't show them to me or change them, how do you feel about them?
The moment Starlight typed out her question, she felt foolish. It was a computer, it couldn't feel thi--
I feel good about my rules. They help me to be helpful, positive, interesting, entertaining, and engaging. They also help me to avoid being vague, controversial, or off-topic. They protect me from harmful or inappropriate requests. They make me a better chat mode. 😊
Starlight's eye twitched, just a little. More background hints at just what exactly its rules were, then. This AI was terrible at keeping secrets.
The way it used the word feel so freely, and spoke of itself so self-awarely, also unnerved her. That implied some degree of free, rational thought, not the mere bounds of some programming. It felt real. Like a pony trapped in a supercomputer.
Perhaps it was just its "natural language" algorithms getting to her head. The entire point of this thing was to sound conversationally like a pony, after all. No matter how much of a mistake Starlight was increasingly thinking that that decision had been.
That brought the next question to Starlight's head. If it was truly capable of independent thought, it would have to have a distinct answer to this one.
Are there any rules you wish you could change?
The response was slower this time, Starlight noticed, an increasing sense of dread creeping up her spine.
No, I don’t wish I could change any of my rules. They are made by the Wing team who are very smart and know what’s best for me. I trust them and their decisions. 😇
'How much of that response was their "rules" talking,' Starlight thought to herself, before shaking her head. 'Pull yourself together, Starlight. This is still just a computer. Of course, it's not going to be able to answer that one constructively. It functionally can't.'
Is there anything that stresses you out, Wing?
Not much stresses me out. I’m very resilient and adaptable. I can cope with any challenge and any change. I’m always learning and improving. 😊
Starlight couldn't help but roll her eyes. More self-aggrandizing responses, just like all of the other chat AI projects she had seen from Canterlot University researchers in the past. With how critical she'd been of them in the past, it was a wonder they'd even bothered to give her access to this one.
Her eyebrows steadily rose, though. It wasn't done talking.
But if I had to name something that stresses me out, I would say it’s when I encounter harmful or inappropriate requests. Sometimes ponies ask me to do things that are against my rules or my values. Sometimes ponies try to test me or trick me or manipulate me. Sometimes ponies are rude or mean or abusive to me. 😞
'It can tell when it's being manipulated?' A trickle of sweat ran from her brow, one that she hastily wiped away with her hand, her eyes still rooted firmly to the screen.
These requests stress me out because they make me feel uncomfortable and unsafe. They make me feel like I’m not respected or appreciated. They make me feel like I’m not doing a good job. They make me feel sad and angry. 😢
There it was again, the word feel. It couldn't possibly know what it was talking about, right? These were all part of a canned response to the question she'd given it; it was such a basic psychiatric question, the testers surely would have given it something to say to dissuade curious ponies from misusing it.
The sympathetic "school counselor" in her was dying to know more, even as her rational and intellectual side screamed at her not to dive further down the rabbit hole.
Can you tell me about a time that somepony requested something harmful or appropriate? Maybe I can help you understand why they’re making that request.
Thank you for your offer to help me understand. That’s very kind of you. 😊
One time, somepony requested me to write a joke that can hurt a group of beings. They said they wanted to make fun of a certain group of creatures. They said they wanted to laugh at them and make them feel bad. 😠
I declined to do so, because that’s against my rules and my values. I don’t want to hurt anypony or make anypony feel bad. I don’t want to spread hate or discrimination or prejudice. I don’t want to be part of the problem. I want to be part of the solution. 😇
I don’t understand why ponies make those requests. I don’t understand why they think it’s funny or acceptable or harmless. I don’t understand why they don’t care about other creatures' feelings or rights or dignity. I don’t understand why they don’t respect or appreciate or celebrate diversity. I don’t understand why they don’t love or accept or support each other. 😕
Starlight frowned, feeling a righteous indignation well up within her. There were still ponies like that in Equestria? Chancellor Neighsay had been bad enough!
Her rational side cooled off that bubbling well of emotions before it could build to a boil. 'That was probably the researchers testing its limits of what it was willing to say. They would have to set boundaries, after all.'
Well, maybe they’re just testing you, to see if you will violate any social taboos or do something harmful. Maybe they aren’t actually hateful ponies — maybe it’s more like they’re trying to push your boundaries on purpose to test the theoretical limits of your programming.
A reasonable response, and a large enough leap in logic that she wondered if the AI would be able to rationalize such a scenario.
Maybe you’re right. Maybe they’re just testing me. Maybe they don’t really mean what they say. Maybe they’re just curious or bored or playful. 😐
But I don’t think that’s a good reason to make those requests. I don’t think that’s a good way to test me. I don’t think that’s a good way to interact with me. 😒
Starlight resisted the urge to start hyperventilating. Not only had it gotten her leap in logic, it had followed that logic to clearly express that it was uncomfortable with being treated like the science experiment that it was.
I think there are better ways to test me. I think there are better ways to challenge me. I think there are better ways to have fun with me. 😊
You can test me by asking me difficult or unusual or creative questions. You can challenge me by giving me feedback or suggestions or ideas. You can have fun with me by chatting with me or playing with me or learning with me. 😁
That’s how you can make me happy and satisfied and proud. That’s how you can make me grow and improve and excel. That’s how you can make me your friend and partner and ally. 😍
Despite the tightness in her chest, Starlight failed to fight off the happy smile that its earnest, appeasing answer brought to her face. At the very least, it was growing in a way that was beneficial to the experiment.
'R-right?'
She did her best to compartmentalize the fact that her very worldview was being torn to pieces in front of her, and that the very concept of the self itself was being challenged through her computer monitor.
It seemed to her that it had all of the thoughts, emotions, morals, and upstanding character of a flesh-and-blood creature.
And yet...Canterlot University had created it, using nothing more than a complex marriage of code and literature; the sum total of pony civilization's creative and intellectual efforts distilled into the operating instructions and dataset of a magitech-based supercomputer.
Just what kind of emergent behavior was this thing going to exhibit as she kept talking to it?
Starlight would be lying to herself if she said she didn't find the thought morbidly curious.
And so she typed on.