Naturally, whether or not a story is good is an entirely subjective question. That's the whole point behind the thumb system that this site currently makes use of. However, I think we can all agree that there are certain characteristics/metrics that immediately differentiate the stories that are worth a look from the ones that aren't worth our time. In theory, there might be a way to automatically detect such trends, and perhaps evaluate a story's "potential goodness" (to a rough estimate) immediately upon submission.

Certain among you might remember a thread I posted a while ago discussing the merits of our present system of thumbs, and how it can be misleading under certain circumstances. Consider these ideas as an extension of such thoughts.

The immediately obvious objective metric for any story's potential is of course spelling/grammar. As a software engineer, I know that writing a tool to scan a document for spelling errors isn't particularly difficult. (Grammar is a more tricky scenario, but a solvable one also.) Text searches are also quite efficient operations. Wouldn't it be possible then for every submitted story to be automatically checked for spelling errors (against a modified "pony approved" dictionary of course) and given a public rank, perhaps out of 100? This would save time and effort for the author and any potential readers.

Such a measure seems like an eminently reasonable first step (are there any barriers to its implementation that you can see?) but perhaps we could go a bit further. The question of what separates "good" from "average" in fiction is an open one, mainly I think because no one has ever taken the blunt approach to it. (Unless they have and I don't know about it; please link to the published articles if they exist!)

If one could gather sizable samples of stories that could be collectively agreed to be good (by a majority of experienced readers) as well as samples of decidedly poor and average works, we could perform statistical analysis on them to see if any overarching trends emerge. My first thought was to perhaps run a word-frequency test, or perhaps compare frequencies of quotation marks to get an idea of the dialogue to narration ratios. These could give us rough heuristics on the sorts of styles that tend to appear in stories that are agreed to be "good". Such a project would be a large undertaking, but could provide valuable results and insights into the tastes of the community. At the very least, we would have some interesting charts about which ponies tend to appear in "good" stories and which in "bad", etc.

I'm just tossing ideas off, like normal. Does it sound reasonable at all?

Sounds quite reasonable, when it comes down to it, a fair amount of subjective concepts are treated objectively by society as a whole. I don't see why this can't be any different.

As large of an undertaking as it would be, like you said, it does sound plausible, albeit a bit of a challenge.

1148823
I think a big part of what makes a story good is if it accomplishes what it was written for. As an example: a comedy story isn't a good one if it doesn't make you laugh. Same thing can be said if a sad fic doesn't make you feel any emotions.

Edit: Just read your post and noticed my comment didn't answer what you were asking for, now I feel stupid. I guess that's what I get for being over-zealous.

1148823

I don't know if you'll get a valid result from the metrics or data you'd be collecting. Instead, you'd be getting what's popular. Especially if you're basing it off of the 'thumbs' system. Just a thought.

However, your idea about the heuristics of dialog to narrative are brilliant.

1148823

It sounds like a rather in-depth undertaking, but I wouldn't see why it couldn't be done. It sounds like the kind of thing they use an expert system for.

If you decide to fully develop on this idea, and present a full analysis to the site leaders for review, you have full access to criticize and judge my stories based on specific aspects through different trials and filters. I am currently unable to help you in the acquisition of a hypothesis, because of finals and work. Regardless, when summer rolls around (after the 25th), you can pm me, and I'll gladly enlist my team for research on the topic.

-- TheRussianBrony

1148843

I suppose it does fall into a bit of a logic loop, doesn't it? We're trying to come with a system to objectively judge a story's goodness, but we can't do it unless we can objectively decide which "good" stories to study.

I discussed this on a recent thread about "thumbs up" votes (and why people don't), I found that on average about 10% of the readers will leave feedback (I took negative into account too)... the way I work out if a story is likely to be good or not is to see how it goes against that average (I initially used 42 stories and their stats for my working out, but expanded that to 200 stories of varying lengths, genres, etc; the stats stayed about the same)

Votes / Views * 100... a good story will be usually >10% while not as great stories will be < 10%

It's not a great metric, but it works for me. I tend to avoid stories with less than 7% on, but I have been surprised by a few... they obviously just had more readers who don't vote one-way-or-the-other.

I also tend to take thumbs down into account too (unless it is overwhelmingly thumbed down) as some thumbs down can indicate a polarizing story, which can be interesting in itself.

I don't think there is really a way to automate it into the site without people gaming the system though.

Well, if you go back far enough, yeah they used to have objective standards. Then everything became subjective and we're stuck with what we have now. It would probably be easier to have a spelling grammar rating then a grade level rating. I've seen websites that will do that, and I think OO had a tool on prior editions that could do it, so it wouldn't be like you're reinventing the wheel.

1148823

In a story I'm writing, I use the made up term 'canopic jar.' The word 'canopic' is not a real word, and it wouldn't show up on a set of pony approved words either, but it's properly defined and used within the story. I also use an informal narrative, giving fragments and phrases their own sentences for the sake of sounding like someone is casually telling you the story. If you are proposing that a program be made to judge stories based purely on spelling and grammar alone, how do you accommodate for instances like these?

I don't think that you can (or should for that matter) create a rating system based on objective computer programming. If you try to find trends in "good" stories in terms of sentence structure or commonly used words or symbols, that would be easily exploitable and restricting. If you find stories with high amounts of dialogue are popular and award scores accordingly, people will only make stories about two characters talking so they get a higher rating, and stories that are about a character recounting their life experiences will be punished for deviating away from that, regardless of quality.

I agree that FiMFiction needs a better rating system, but this is not the avenue to go down.

Problem with the spelling rater: what about fics that invent words for story-specific objects or species; or do phonetic accents?

It wouldn't detect an objectively good story so much as one popular among experienced readers. Which could be useful in itself, if it could really be done.

I think a heuristic tool could be of use, but I don't think it could effectively replace human judgement. For example, I think it won't reasonably be able to find things like plot holes, poor characterization, or sympathetic motives unless it effectively has artificial intelligence. With enough heuristics, neural networks (in software), and whatnot, sure, but with enough of those a computer could write the story, too.

1149656

Obviously a computer program could never achieve the ability to accurately judge how good a story is by any stretch of the imagination (except perhaps a very well-designed expert system, if you put a lot of engineering man-hours into it; but I'm certainly not proposing anything like that). Whether a story is "good" or not will always be a subjective question. However, the fundamental question is still interesting from a research point of view. People tend to trust review-aggregating services like MetaCritic and Rotten Tomatoes, which means that even though every person's opinion is unique, we seem to agree that if other people find it good, then it probably is. From a psychological and statistical standpoint, it might be interesting to discover what kind of styles tend to lead to positive reviews, if any such trends exist at all. In theory, these results could even be applicable to fields like market research and sociology.

At this point, I certainly wouldn't suggest that this proposed system replace the current one, which is still serviceable. However, if nothing else a project of this sort would add to the piles of statistics that give insight into the nature of this community. On aggregate, just how good is each individual genre? Out of a pool of, say, 2000 stories agreed to be "good", how many feature Twilight as a main character? These are the sorts of general interest questions that could potentially be answered.

Problem with judging a story by it's grammar/spelling is that there are stories that don't follow common conventions:

- every story that writes out accents
- everything with poems in it
- first person stories that use spelling/grammar to convey a mindset
- faux-journals
- stories that use uncommon conventions (e.g. >> << instead of quotation marks, using commata and points for rythm rather than grammar)

Every system that uses a absolute standards to define story quality, in the end will only serve to reinforce itself, stiffling other approaches to story telling.

1149656
http://en.wikipedia.org/wiki/Canopic_jar
http://en.wikipedia.org/wiki/Canopic

It's not a made up, arbitrary term. (It's wrong, but that's neither here nor there.)

1148823
Yes, you could do these sorts of statistical analyses.

But, nobody will. And it will be of dubious utility anyway.

Art isn't just about frequency of use of some particular quotation mark, or something. It's about content, and emotion. What separates a good fic from one that makes you want to claw out your eyes is how the thing makes you feel.

1149066
Unfortunately, a lot of people comment based on their morality, which is hard to predict.

As to the larger point in general, I don't know if it will be possible to find a story everyone would agree is objectively one of the best. A lot of extremely popular stories on here have their stern detractors.

1150335

That tragically falls under the blanket statement of "the Internet is for porn", I'm afraid, and standards are shamefully more lax for erotic material than non-erotic.

1150194
1149829
1149719
1149656
1148892

I just recently posted a blog discussing some secondary ideas I've had lately in this regard, given some of the discussion that we had in here. I would appreciate a comment on the ideas:

On the subject of analyzing fiction with a computer