Reviewing AI written pony fanfic reviews because I am in hell now I guess · 9:26pm Jul 26th, 2023
Definitely posting this against my better judgement, but recently, I keep seeing this one group linking to AI written reviews in people’s comment sections. And, honestly, I’m kinda fascinated/horrified by these things? Mainly, though, I’m alarmed at the thought of new authors being spammed with this sudden deluge of ‘objective’ writing advice – as of the time of posting, almost 110* honestly-quite-lengthy reviews in less than two weeks.
*Actually, sorry, my bad, more like 50 -- I misunderstood something. Still kind of outrageous number to me though tbh
DISCLAIMER: I am NOT naming the group nor any of the fanfics in this post, I DO NOT encourage people to go looking for the group, and, to be abundantly clear, this is NOT a personal attack against any of the group’s members, none of whom I know and have nothing against. But if this group feels comfortable offering unsolicited writing advice to people (review submission folder states ‘Do you know of a story? Post it here!’), I can do the same to them.
Let’s start with this little example from a review of a short horror story. When asked ‘What are the best qualities of this story?’, one of the bullet points the AI responds with is:
- Haunting ending - Leaving Twilight's fate uncertain closes the story powerfully and disturbingly.
Shortly afterwards, when asked ‘What are the worst qualities of this story?’ it answers by saying:
- No concrete ending - Twilight's ultimate fate is left uncertain.
So, the ending is both powerful and haunting because of the intentional uncertainty, but it is also bad because of that very same uncertainty? Got it. Crystal clear objective feedback.
Let’s look at another review, this time for a slice of life story. The AI is told ‘This is a completed one-shot story’ – not a question, just a statement of fact. The story has an ending. Literal toddler level stuff.
Naturally, the AI responds to this non-question by saying (and I’m gonna remove the character names to make the story a little more anonymous):
You're right, after reviewing the full text it is clear this is a self-contained, one-shot short story that has reached a conclusion. The key evidence that it is complete:
- It has a clear narrative arc - the story follows [character] from recounting her past up until the resolution where she finds acceptance and support from her new friends. This provides a satisfying character journey with a beginning, middle, and end.
- [Character]'s backstory is fully revealed through her retelling, providing complete context. We learn important details about her motivations and experiences.
- The core conflict is resolved emotionally – [character] finds belonging with her friends who promise to stand by her side, addressing her concerns about losing their friendship.
- The story comes full circle - it starts and ends with the friend group together, bookending [character]'s retrospective narration.
- Loose ends are tied up - Questions around why [character] left home and sought out the other ponies are answered thoroughly.
- The ending provides closure – [character] going to sleep comforted knowing she has the support of friends as she prepares to face her past sums up her arc.
So in summary, the character development, backstory details, resolution of [character]'s inner conflict, and the contained story structure indicate this is a finished one-shot rather than a continuing narrative. The author provided a satisfying reading experience within this self-contained tale.
Yes, thank you, AI, thank you very much for taking almost 230 words to explain that this story, indeed, has a beginning, middle, and end. Insightful stuff. If I was an author receiving this review, I wouldn’t feel at all patronised 😐
Let’s check out another one. Here’s a response to ‘What are the best qualities of this story?’
- Subtlety - The tragedy unfolds organically without feeling forced. There is an understated quality to the writing.
And then here’s a response to ‘What are the worst qualities of this story?’
- Subtlety - Some of the emotional elements verge on melodramatic. More nuance and restraint would boost credibility.
So, in the pro column, we have ‘Subtlety’, and in the con column, we have ‘Subtlety’. A wonderfully subtle story with beautifully understated writing, and which is also melodramatic, unrestrained, and which requires more nuance to ‘boost credibility’.
Specific examples aside, it’s the larger, more general shape of these reviews that concern me – the vagueness of the AI's praise, and, more importantly, the aspects it so often picks out as negatives, a lot of which are frankly… strange? Odd details to fixate on, especially in the context of fanfiction? As though the thing writing the reviews isn’t, y’know, actually human, or has any real understanding of what makes humans tick?
- The AI does NOT UNDERSTAND that slice of life stories don’t require high stakes. That the lack of high stakes is often kind of the darn point
- Not every single character in every single scene needs to be fully fleshed out. Background characters are just that – background characters. They belong in the background
- Similar note, but if I’m reading a pony fanfiction on a pony fanfiction website, it’s a good guess that I probably know who all the ponies are ('Lots of name dropping of MLP characters which may be hard to follow for non-fans.')
- It is OK for stories to be short
- Seriously, it is ABSOLUTELY OK for short stories to be short. Making short stories longer doesn’t automatically make short stories better, AI
- It’s OK for plots to be cliched sometimes on a fanfiction website. People like the cliches. I like the cliches. They’re fun to read, fun to write. A cliched story – well executed, and spun through the lens of the author’s personality – can be a helluva good time
- Similar point, but it’s OK for stories to be predictable sometimes. Being predictable is not inherently a negative
- Not every story needs a subplot
- For real tho, not every story needs a subplot
- The AI has an obsession, too, with settings being 'underutilised', but without ever explaining what it actually means by that. Once again, a weird thing to fixate on in fanfiction imo, where most people are already very familiar with most of the settings, and are mainly here for juicy poni drama. (The worst was the criticism that stated 'Limited settings - Most events happen in enclosed interior spaces, lacking variety.' Because, as we all know, no good story has ever taken place indoors.)
Really though, all of these kinda boil down to the same problem, which is that the AI so consistently misses the point of fanfiction that, if I didn’t know better, I would swear it was doing it on purpose.
But how could it NOT miss the point? How’s the AI supposed to comprehend the many, MANY different reasons that people are drawn to writing fanfiction in the first place? How is the AI supposed to know that some people do it to blow off steam, or to have fun exploring a concept or idea that got stuck in their head? How’s the AI supposed to know that others pour their hearts and souls into their work, and that raw passion isn’t something that can be accounted for with a simple set of number ratings, or with a Twilight Sparkle-esque checklist of pros and cons? How is it supposed to understand that all these different approaches to fanfiction are 100% valid? How is it supposed to understand any of this, any of this, when it doesn’t even understand the very meaning of the words it’s reviewing, when all it understands is simply what word is most statistically likely to follow another? How on Earth could it possibly comprehend the joy of fanfiction, the thrill of finding others who are just as obsessed with your own particular special interest as you yourself are?
The answer is that it can’t – it can’t understand that there’s more to a review than just listing the simple technical components of a story. And so it’s no surprise that it always misses the point. It’s no surprise that the rough edges, the little quirks and uniquenesses that make so many stories on this site engaging are reduced to a weird little list of cons – errors to be shaven off, smoothed down until all that’s left is the dull grey dreariness that AI likes best, average stories with no taste or flavour, no personality, no anything.
I’m certainly not saying that you can’t point out flaws in a story, or that authors can’t/shouldn’t strive to get better if they wish to (heck, that list of negatives in the section above? In the right context, any one of those could be a valid critique of a story). What I am saying, though, is that I wouldn’t trust an chatbot to review a story, no more than I would trust my calculator to explain all the themes and nuances of Fallout: Equestria to me.
And, look, I mean… I know, I know that I’m shouting into a void here, that I’m getting myself worked up over a tiny little group who, whilst I don’t like what they’re doing, seem to be having fun + abiding by the rules of the site. But small though it is, it still adds to the dread I’ve experiencing all this year – another example of AI’s slow creep across all the sites I love, all the places online that mean something to me: a few pictures here, a couple of stories and articles there, almost all of it soulless, boring, dull, dull, dull. It’s like… this is the future people want for fanfiction? This? This??
Reviewing is an art form. One astute, constructive review can be enough to change the course of your whole writing career (thank you Vimbert, wherever you are), and a good review can stay with you forever – heck, years and years after I printed it off, I still have PresentPerfect's review of In the Place the Wild Horses Sleep over my work space. It means something. It means something, because it came from a person – someone who brought their own thoughts and experiences to my work.
If you are wondering what's up with 'PRINT ON LONG EDGE', it means that I am Extremely Good at using my printer
By definition, an algorithm can’t do that. It has no thoughts, has no past experiences. And, for the purposes of reviewing someone else’s story? Well, that’s pretty much a disaster.
My work's had some savage reviews on times – all of them valid, all of them useful to me. But, man, when I was starting out here, it would’ve killed any enthusiasm I had to have some cold uncaring AI review assign an ‘objective’ 6 out of 10 to my work (because, jeez, it is almost always either 6 or 7 out of 10). Legit, I think I might have just given up. Wild Horses certainly wouldn’t have existed if I knew it’d get feedback like this, that’s for d*mn sure.
ANYWAY, sorry, this ended up being at least a thousand words longer than I meant it to be (can you tell I feel passionately about this lol?). So, summing up, I would rate these AI reviews 6 out of 10, room for improvement, writer appears to be of intermediate level.
Here is my breakdown:
Pros:
- Knows that stories have beginnings, middles, and endings. Excellent work, AI, I am extremely proud of you (10/10)
- I like that you spell things good, AI, this is a positive (8/10)
- Correctly recognises that the best stories are the ones that don't take place indoors (7/10)
Cons:
- I, too, am very smart and know that stories have beginnings, middles, and endings. I promise you don’t need to keep explaining this to me, AI (4/10)
- 6 out of 10 (1/10)
Yeah. I really don't get the appeal of these things if someone is going for genuine advice, and I really hope nobody is.
But, if they're just doing it for whacky fun to see what nonsense the silly robot spits out, eh. Live and let live I guess.
Or at least that's what I tell myself to stave off existential dread, lol.
The hallucinating algorithm does not know what it sees. The overheated processor does not feel the heat of artistic passion. A mindless process cannot comprehend art because art by definition interacts with the mind. Otherwise, it's just noise and blotches.
So yeah, this group deserved this.
The group's concept is astoundingly dumb from the get-go, and those snippets are about what I'd expect from such foolishness.
As FoME said: that group deserves this.
The conceit that something algorithmically driven can produce quality critique because it’s “objective,” and therefore better, than a flesh-and-blood reader, has already taken hold among readers and writers alike. Years ago, I remember seeing this one guy brag about running his story through four different spellcheck programs to proofread it. The story was still riddled with grammatical errors, to say nothing of the prose’s overall quality, but the author wouldn’t hear of it. “I guess you know better than the top four spellcheck programs on the internet, huh?” he’d say, except with less punctuation and looser sentence structure.
This AI shit is an extension of that, and it’s gonna lead a lot of people (budding young writers and talentless hacks alike) down a bad path.
Yeah, but you can write “80085” on a calculator, which is more rewarding than reading Fallout: Equestria, and on about the same level of sophistication.
Yeah, I'm with you and the commenters thus far: This kinda sucks as a concept. I'm glad they're having fun I guess? But it feels so soulless.
Printing out PP's review and saving it like that is such a wholesome, human thing to do. It made me smile, and in an indirect way it strikes me as the exact antithesis of the AI-generated reviews.
Also, this post inspired me to take a stab at getting their preferred AI to do some reviewing for me, and it was very much a "Dead Dove Do Not Eat" experience; I don't know what I was expecting.
first of all, I'm sorry you felt like you had to submit yourself to that D:
second, I am now waiting for the day when someone says basically anything about this garbage in one of my blogs :B heads will roll
third OH MY GOD I'M KIND OF CRYING NOW I HAD NO IDEA THAT MEANT SO MUCH TO YOU I AM SO TOUCHED ;_; and now I have a digital picture of the printed picture of the digital thing, for my own keepsake
The main problem with AI is that there's no intelligence there. ChatGPT is just a glorified autocomplete.
This kind of stuff really bothers me. Not that an AI is capable of doing the above (that's actually pretty cool, ngl), but that people think this is somehow acceptable or the be-all end-all, that people think removing the human element from a uniquely human craft is somehow a good thing.
Art is art because it is human, not because it is visually appealing. Even when (yes, when, sadly) AI gets good enough to effortlessly craft masterpieces, it'll still never be as good as art made by flesh-and-blood people.
This would have absolutely murdered any passion I had for fanfic if I got something like this way back when.
The funny thing is, these "objective" reviews are still very much subjective, since they're still using a prompt from a human to tell the AI what they want it to say. And in its own way, that's even further from the mark of objectivity, because it removes the ability to be objective, because it's an extra step removed from the source material.
P.S. edit: the PP review is well earned, and it makes me happy to see it printed out like that.
This AI nonsense in place of human critique need to stop. It’s lazy and soulless.
The novelty of this new machine has worn down.
Bam. Nail on head moment right there.
Using AI to write for you/instead of you or using it to "critique" something that it can't understand the purpose of to begin with, just reeks of laziness. But as we've seen from comments in blogs and such, there's people that are here just to get content, and they don't care about quality, depth or effort as long as they get something that resembles what they want... I'm not surprised that some people would flock to "reviews" that will tell them nice generic pre-canned things about their stories because it's easier than knowing that you might've written something an actual person might not like.
Eh, I see it more as a curiosity. I added two of my stories to it because I was curious if they could point things out that I hadn't been noticing. Though I think mine got ignored. I don't fully get people's animosity toward AI since it's really not that great at writing stories (it can probably edit them pretty well).
I've seen what it can write and it can't write better than good writers for good films or fics or games. It can, however, write way better than crap writing which is what scares some people. (opinion of mine, obviously, I'm not stating this as a fact). AI-generated ART, oddly enough, is leaps and bounds ahead of what these AIs can write which you would think would be the opposite given the difficulty in drawing over writing (drawing well). But that just goes to show you that good writing really is very difficult to accomplish.
5739558
This would explain why it's so bad at writing good stories.
5739486
You and me both!
5739487
Love this. Beautifully put!
5739500
Thank you for saying this. I was honestly kinda worried this whole blog was just gonna come across as mean-spirited or something? So it’s pretty gratifying seeing a whole bunch of others agree with me on this. Makes me feel like I’m not going crazy.
5739502
Great little anecdote, and this is so much the vibe I got from the whole group. Looking through their forums, everything's layered in irony -- but, like, at the same time, they super give off the sense that, at least on some level, they really do believe what they're saying? That the AI really is a fair and objective critic, rather than something that's been trained off ungodly amounts of human written text, along with all the inherent biases that brings with it
Touché!
5739503
I started off thinking so too, but honestly, the more time I’ve had to think about it, the more uncomfortable I feel? It would be one thing if it really was just them having fun amongst themselves. But it's the fact that, looking through their forums, I keep seeing comments from authors who seem to be taking the reviews seriously, combined with the fact that you can add stories to be reviewed without asking the author’s permission first. Like Posh got at below, feels like it's leading people down a bad path
5739506
Oh wow!! I can't look at this properly right now (I'm in work), but I will definitely look at it later
5739517
Honestly, it really does. It's been such a source of motivation over the years, especially when I feel down about my writing (which is quite a lot tbh). Reminds me that I can do it + never fails to pick me up
5739558
Amen to this!
5739567
Agree with all of this, it's really well said. Certainly can't deny how impressive the technology is -- downright miraculous, even. But seeing how a lot of people are actually using it is, whilst not unexpected, still utterly dispiriting. And that's not even touching on how, time and time and time again, it feels that the people most enthusiastic about it online also seem to be the ones least unwilling to learn how it works + the ones who absolutely can't understand that AI is not in any way objective
5739569
For real though, even with my low expectations from the AI, it was kinda shocking to me how often it made this specific complaint of SoL. Like, to be fair, not on every SoL story? But certainly on enough of them to raise an eyebrow 🙃
5739602
How on Earth you and the other mods dealt with the comments on that blog post the other month without ending up banning half the users there... legitimately heroic effort + I think I would’ve given up after the first 100 comments. Very genuinely, you should be really proud of yourselves
Totally agree, and getting pre-canned positive feedback definitely seems to be a big part of it. But honestly, I feel the full reality is also a little stranger? Poking around in the group, some people really did seem to value the pre-canned negative things as well, treating the AI’s feedback as inherently better than anything from an actual human because of 'something-something-objectivity'. Even though sooo many of the critiques were vague at best, and self-evident nonsense at worst.
5739665
In spite of everything I wrote in the blog + in other replies, I 100% get where you're coming from with your curiosity. The technology is mind-blowing, and that it can do the things it does is incredible!! But if the AI does end up reviewing your stories, just remember to take everything it says with a huuuge grain of salt, especially any negative critiques.
… I mean, it might even end up saying useful? Though if that ends up being the case, it would be because of what you yourself bring to the review, if that makes sense. Like, you're the one with the human perspective in that interaction.
Haha! Spot on!
5739828
Oh bleah. Yeah...that kills it for me. People shouldn't be taking that sort of stuff seriously. Or trying to suggest others should. :/
So, coming in quite late, but we, the interviewers, me most of all, were learning. We've gotten better at asking the proper questions and conducting these interviews to dig at the heart of things better. Sorry for the awkward early review, also sorry because, at first, we didn't have rules about who posted what story. That has since been fixed. Only authors can post their own stories. Apologies once more.