An Accurate Trick to Estimate Fimfic Rankings · 6:35am Jan 9th, 2016
On my user page, I maintain two lists of stories:
Stories of Mine You'll Probably Love
Stories of Mine You'll Probably Hate
Deciding which stories should go in the "Love" list isn't trivial. What makes a story good? I can think of several things:
Enjoyment: a random reader would probably like it
Popularity: the story has a lot of readers
Low Downvotes: the story is not controversial
These aren't the same thing by any stretch. Back to Normal is one of my most popular stories, but it's also extremely controversial. I'm most interested in capturing Enjoyment, but Popularity and Low Downvotes are conflating factors and it isn't immediately clear how to balance them. For example, a story with 20 upvotes and 0 downvotes has a 100% like ratio. But if it only has 20 upvotes after a long time of sitting there, it's probably not as "enjoyable" as a story with 200 upvotes and 20 downvotes.
In other words, stories with a low vote count should have lower scores even when the ratio is high. Originally, I was already applying a penalty to my low-count stories in order to account for the uncertainty of a small number of votes: if the next couple of readers were to downvote a story with a low vote count, it could radically alter the proportion of upvotes to downvotes. In contrast, stories with more votes would be less affected by a little noise.
But then I started to realize that even if the count were accurate, those stories should be worth less anyway because they're less popular. Back to Normal is more enjoyable than its vote ratio implies because the controversy is part of its appeal. Stories that please, but fly under the radar, aren't as interesting as those that draw in a large number of readers.
Yesterday I rediscovered the story ranking metric here on Fimfiction (to see your story ranking, you need to click on the statistics button from the main page of the story). Story rankings on Fimfiction probably take a lot of factors into account, including when the story was published, number of views, number of favorites, upvotes and downvotes, and so on.
I decided to chart the rankings of my stories to see if I could find an easy way to predict a story's rank from other factors. I quickly discovered something surprising: you can predict a story's rank with amazing accuracy by only using upvotes and downvotes, as long as you weight them properly. I'm sure my data set is not totally representative, but I used the 18 stories I've published (including the two which are currently taken down), and this is how well my formula worked:
I have two fics in the top 5k. The removed Flash Sentry fic is the worst, but still above average.
(There might be < 100k fics open, but there are ~150k if you count the ones no longer up.)
The trick is simple: add four downvotes, then use the vote ratio. That's all there is to it. It might seem like adding the downvotes isn't a huge deal, but it is, because it significantly reduces the scores for stories with low vote counts. I tried using views and time since publication and other metrics, but adding four downvotes was eerily accurate for my sample set. I also tried to fine-tune it with fractional vote counts, but it worked best with exactly four. (Before fine-tuning, I was using three instead of four, so I almost had it right with my original guess!)
Of course I'm overfitting by fine-tuning to a dataset with eighteen points, but I'll wager this would give you a pretty good rule of hoof for comparing fic quality for your works, too. I'm curious if anypony else can confirm my theory with more fics. I'm too fatigued at the moment to start looking at other ponies' fics' stats. Finding a rule for my fics in particular was why I did this, and it definitely works for that.
I've got a pretty large pool of fics. If you feel motivated to use them as a sample set, have at it.
That's so weird. Very interested in seeing where this goes.
According to Titanium Dragon, the actual formula is a bit more complicated, and involves looking at the lower bound of the confidence interval for an estimate of the fraction of upvotes a story will receive:
https://www.fimfiction.net/blog/372794/fimfiction-rating-system-explained