The More Most Dangerous Judging · 3:14am Mar 30th, 2015
One day, I was minding my own business, procrastinating on writing as usual, when the false prophet Obsolesence crawled out of the depths of Hell and asked for my hand in carnage. I asked him for the terms and conditions, at which point he produced a stone tablet writ with the blood of deceased presidents. After coming to the conclusion that no, I did not have any tributes within immediate reach to volunteer, I accepted his offer. I soon found myself shackled with several other misguided souls, and the rest is history.
Or something to that effect.
So in case anyone’s curious about how the judging for this contest actually worked, what we did was adopt the judging system used for Outside Insight (CouchCrusader was graciously brought on board for guiding us, and Themaskedferret was an excellent source of moral support.), rather than reusing the first TMDG system of every judge individually reading every story. This was much appreciated, since there’s no way I could bear reading a whopping ninety-three entries. So what we did first was divide the twelve judges into groups of three.
Round 1: Screening
This was fairly straightforward. There were four groups of judges, each a group of three, and the entries were divided up evenly between them to ease the overall burden (each group ended up with twenty-three stories, give or take). They would then vote on whether or not the story would pass to the next round. The stories themselves during this round were largely skimmed through, I think. I know a judge or two read every story completely, but since this was the screening round, all we had to do was vote on whether or not the story had a chance of reaching the winner’s circuit. So not everyone read every story completely. While I read a couple end to end, I know I only read the first chapter of some, while I skimmed several others, and I think this is largely the case for everyone involved.
The kinds of stories we were specifically looking for were ones that didn’t fit the spirit of the contest or broke the rules (we did in fact get a couple submissions that had nothing to do with the contest whatsoever), or stories that were too poorly conceived, written, or simply just not good enough to have a chance at reaching the end. Most of the stories ended up falling into the former class, although there were a few in latter.
My group in particular turned away thirteen of our twenty-three, and that ended up being the case for the other groups as well, give or take a couple. What I found interesting was that all the stories my group turned away or approved were all agreed upon—by all three of us—with one or two exceptions. I can’t vouch for my fellow judges, as I haven’t got access to their sheets at the moment (nor do I care to peruse them), but the stories denied at this stage by my group were overwhelmingly bland. As in, they aspired to not try anything with the prompts and more or less stuck to the tones of the stories that inspired them.
Basically, if your story had a unique spin to it that was more than substituting Rarity for Rainbow Dash in Cupcakes, it moved on to the next round. It took us about a week to do this, in part because of the varied amount of time people could dedicate to judging. But a week had come and gone, and in the end, we reduced that initial ninety-three all the way down to a comfortable forty-twoish.
Round 2: Judge Harder
Now this is where things really got serious. The forty-something stories were randomly divided up again with the same judge groups, the only catch being that no judge would see this round any story they saw the previous round. It was basically the same setup as the first round, and the aim here was to rinse and repeat this process until a comfortable number of finalists could be reached. Fortunately, this only took one round.
Like I said, things were more serious. We read every story completely, gave them greater scrutiny, and judged these batches more harshly with the goal of picking out the stories with a serious chance of winning: the cream of the crop. I don't know about the others, but in my group, we wrote little blurbs alongside the stories we had to help express our votes. And we only passed three, and those denied were all completely agreed upon. Which was nice.
I don't think I can overstate the fact we had a healthy number of judges with a varied degree of taste and preferences. This allowed us to weed out potential biases and also made for a fierce discussion when it came to organizing the winners. There were several stories that ended up making it into the next round which I greatly disliked, one in particular passed by my group which I surely expressed myself against. But hey, two yays beats one nay. Diversity is good.
All groups combined, we ended this round a week later and had a total of fifteen finalists to show for it.
Round 3: Finally Getting Somewhere
With this, we had a workable number of finalists. Our goal now was to make sure we read all these stories and then rank them appropriately out of 10. The 10 went to the favorite, the 9 to the next in line, and so on. Of course, this meant five stories were stuck with a 0. Every judge did this, the points were added up, and in the end we had our ten winners. This, again, took us about a week.
Round 4: The Final Countdown
This round should have been fairly easy to do, but for some reason or another it took longer than it should have and we finished at the end of the month, a few days behind schedule.
The ten highest rated stories from the last round were simply bumped up, filling any voids from finalists not making the cut, and then given appropriate scores on the same ranking system. And with that, we had our winners.
Round 5: Not Actually A Round And More Of An Afterthought
It’s no secret to the other judges that there are stories that ended up being winners which I don’t think were worthy enough. I also think that this is largely the case for the other judges, seeing stories they didn’t like reach the final ten. But hey, I already discussed the importance of diversity and what not.
But I can’t help but think how this contest’s judging system could be improved. I think having such a large number of judges is beneficial when dealing the initial workload, but I think we could’ve been more efficient with three groups of five, rather than four groups of three. There’s always that chance two judges could really like a story that all the others would hate, and a larger initial reaction to said story would help ensure the overall popular vote would stay, well, popular. A part of me is of course going to believe that was the case here (How in the world did Twilight’s Fur Isn’t Lavender not make it into the final round, but Blight of The Salmon-horse did?), and I wouldn’t put it past the other judges for thinking like this too. In fact, a few have already voiced their opinions on the matter—myself included. For a taste of that, check out the Honorable Mentions list.
Another idea I’ve had post-contest was to group the judges and the stories they see based on the judges’ tastes. I know this seems counterproductive to the idea of randomization, but I think it would help that the romance stories be seen by judges who like romance stories, as they would know more what to look for. Perhaps not during the screening round, but during the more scrutinizing secondary round leading up to determining the winners. One round of screening, another of picking the finalists, and then however many rounds necessary to vote on the finalists (assuming a healthy dose of finalists).
The judges could get three weeks to read the finalists and vote on, say, their Top 20 of like 26. Then you vote on your Top 15. Then your Top 10. And bam, winners. This could narrow the screens, so to say, so stories that were good enough to pass the screening round weren’t wrongly thrown out after the next round because two judges, of a group of three, out of twelve total, didn’t like it. Of course this would cause for a more careful selection of the judges themselves, so we don’t get stuck with nine out of ten who don’t like romance, or who all know what to look for in a darkfic but only a few know what to look for in a comedy.
Or maybe the judges themselves could be evenly distributed into pools according their story preferences. I dunno. To the best of my knowledge, TMDG2 is only the second contest to utilize this judging method, the first being Equestria Daily’s Outside Insight. Only further experimentation and tweaking of the variables will be able to make a really good system that balances time, workload, and effort, while also managing potential biases and other quirks among the judges.
Something else about this particular contest that I’m undecided about is the prompts themselves. Early on, as the judges were being rounded up and Obs presented his prompts, there was already disagreement brewing. The My Little Dashie prompt initially forced the author to use Rainbow Dash. The Past Sins prompt was initially restricting the author to use Twilight and Nightmare Moon. We couldn’t really improve the Fallout: Equestria prompt to allow greater creative freedom, and I think we struck out on trying to improve the Anthropology and Cupcakes ones.
Of the prompts, three were based off notably dark stories (only three of the ten winners weren’t dark, and one of them was sad), which resulted in most of the entries being some sort of dark. I would have much rather swapped at least one out based on a story that was more lighthearted, but no agreement could be made on that front either, and there weren’t many ideas tossed around, besides.
While I think the intention of the contest was a well-conceived one, the ambiguity of the prompts definitely led something to be desired. Creativity pulled through in the end, seeing the number of great finalists we had, but I think that number could’ve been higher and more diverse. There was some disagreement among the judges initially about the spirit of the contest as well: should entrants be told to follow a story’s general plot (the prompts), or the spirit of its themes and overall tone (the directions)? It ended up being an ambiguous mix of the two, which showed to be a little confusing on both fronts.
Perhaps the contest’s subject matter was too restricting, then. TMMDG certainly had less openness than its previous iteration, and while the turnout was higher (which can likely be attributed to the signal boosting by Equestria Daily), I think there could’ve been more. So if there’s anything I would change or advise for future contests, it’s to make sure the prompt(s) are vague enough to be spun in several directions. Such restriction as experienced here can be fun, but it does narrow the potential for creativity in my mind. Oh shit. Now I’ve got an idea for a contest... >_> The Even More Most Dangerous Game?
I also can’t say how the other judges ranked the stories for when it came to deciding the winners, but I at least stuck to the tried and true method of ordering by reading experience. Basically, if I enjoyed story X more than story Y but less than story Z, I ordered them as such, and then applied the numbers to them. As with any story, I can usually disregard certain hindrances like mechanical errors, awkward characters, or whatever else that’s problematic with a story, as long as I enjoyed the overall reading experience and they weren’t glaring issues. That’s what matters most to me. However, there’s no escaping that those factors do play roles is making a story enjoyable, and while they did influence my judgement, I tried not to let them get in the way of the story itself.
I agreed to be a judge because I had no idea for a story I could participate with. Figures I would get an idea for an entry right after the thing closes, but whatever. Another story for another time, I guess.
This ended up being much longer than I intended. I apologize in advance for any following flame wars.
It's always interesting to see the backstage action of something like this. Thank you for sharing.
Guilty as charged!
I don't feel that this is that great an idea. I feel that the "what to look for" aspect, when you're getting into something as specific as genres, is probably going to wind up being too subjective to use as a useful basis to decide which judge is given what stories to look at—some amount of extra time would be spent evaluating who likes what genre when, ideally, your judges should be able to handle anything (or at least anything that would come their way in the given contest). To use myself as an example, I enjoy reading romance stories, but I couldn't say that I know better than the next guy "what to look for" apart from the things that I'd look for in any story and the trends in romance that bother or excite me personally.
There's some irony in my saying this, considering that we use a preferences system just as you described when delegating stories for review at WRITE, for the reasons you put forward—that we hope people who tend to like (or at least who don't dislike) a given genre of story will be vaguely better equipped to review a story in that genre. I just can't stretch it to judging, though, given that you're talking about grouping people based on these preferences (which would probably leave some genres with almost no assigned judges and/or have some judges judging a disproportionate number of stories, or some such thing), and that if a fic makes it through to the end of the contest, everyone's going to have to be able to judge it anyway.
2923399
Valid point, but I also brought up distributing the judges based on their preferences. For instance, each grouping gets a person who really likes romance. I don't know what sort of effects either scenario would have, but they seem like things to consider.
BECAUSE YOU ARE WE--
No, seriously, no one should have to read that many stories. ._.
Interesting.
2924121
You are the reviewer Fimfiction needs, but not the one it deserves.
2924315
Who does it deserve? :B My guess is TD.
2926214
I think JohnPerry does/did a fairly decent job.
2926356
oh :(