The HORSE feedback system (and story tiers) · 12:25pm Jan 5th, 2015
I have released this system under a Creative Commons Attribution-ShareAlike 4.0 International License.
(That means I want you to use it yourself if you like it — just give me credit for the idea — and to modify it and re-release it under the same license if you think it could be better!)
If you're reading this, you probably were linked here because I (or another reviewer) gave a story a HORSE rating, or used its associated ranking tiers. The purpose of this post is to explain what the scale can tell you about a story. (Skip to the bottom for the tier information.)
HORSE is basically HITEC v2.0, with a catchier acronym and slightly different categories that I hope will more effectively cover a story's full range of strengths and weaknesses. Here's the original HITEC thread if you're interested in the ways it has developed.
Purpose
The HORSE scale is a chart measuring a story's relative strengths and weaknesses. It is NOT a judgment of story quality: it's meant as a supplement to traditional rating systems. For authors, it can show which areas would be most useful to focus on during editing. For readers, it can give you an idea of whether a story's shortcomings are likely to frustrate you, or whether it might do something worthwhile despite faults you're willing to forgive.
HORSE is a normalized system — which means that every story, no matter how strong or weak it was, gets the same 100 points to distribute between its five categories. This can be done with percentages, like so:
H-45 O-10 R-15 S-25 E-5
Or colored bars, like so (each square is 5%):
HORSE: ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉
Some readers have also suggested making the colored bar out of the letters themselves:
HHHHHHHHHOORRRSSSSSE
The five categories, in order, along with a sample of how I would apply this rating system to fandom ur-story "My Little Dashie":
Hook - Pinkie Pie knows that getting a smile during a first impression can make an immense difference to everything that follows.
Hook is solely about that first impression: the title, the first few paragraphs, and (on FIMFiction) the story description. This may seem like a trivial thing to focus on, but when you're trying to pull readers in from the front page or featurebox, first impressions will make or break you. Did your explanation of the story's core idea, and my first introduction to your prose, make me want to continue reading? ((45): A reader-insert character finds filly Rainbow Dash in a box. MLD pretty much spawned its own genre when it first came out, and let's face it, 500,000+ views is as close to irrefutable as an argument for "this is a compelling idea" comes.)
Originality - Rainbow Dash knows that the best way to show your awesomeness is with the stunt nobody expected.
Originality is a measure of how well the story works with new ideas, or old ideas in unexpected ways. This can take many forms — an unusual core premise; deep and coherent worldbuilding, or alternative character interpretations; novel ways of exploring canon; etc. ((10): A relative weakness. I've long been frustrated by MLD's many failures of coherent worldbuilding; this has been deconstructed in many places I don't feel like taking the time to google. It was the first story to deeply explore some now-overplayed tropes in MLP fandom; it gets credit for that, but hasn't aged well.)
'Ritin - Applejack knows that the fanciest ideas in the world won't help unless you also stay grounded in the basics.
The quality of the prose in the story, both small-scale and medium-scale. Spelling, grammar, punctuation. Colorful language, proper showing/telling, literary devices and metaphors, authorial voice. Choosing words that augment the mood and narrative. This is the category most often critiqued in a story, and the category that's easiest for a proofreader to fix — but still only a fraction of the work that goes into a solid piece of fiction. ((15): Slightly weak, but not its biggest flaw. The language is unexciting, telly, and often repetitive, but nothing in it is poor enough to break you out of the narrative.)
Structure - Rarity knows that, like with clothing, a story must have both its broad outlines and its little accents unite into a harmonious whole.
If 'Ritin covers the trees, Structure is the forest. Does this tell a complete story or paint a self-contained scene? Does the work have a coherent overall theme and/or tone? Does every scene have a purpose that contributes to the overall impact of the work? Is the pacing even, or does it stagger between exciting parts and lulls? This covers not only the consistency of the writing (how much it makes every word count), but also the coherency and the flow. ((25): Relatively strong. Each scene has a place in the overall narrative, and it solidly hammers in the contrast of the narrator's bleak life and surroundings with the joy of Dashie's presence.)
Execution - Fluttershy knows that no matter what talents you might or might not have, what really matters at the crucial moment is simply that you perform.
The strength with which the story's core idea was sold. Did it deliver on the promises it made? Did it inspire the emotions the author intended? Did the characters feel authentic and act realistically, and did the plot flow naturally from their actions? Great execution can redeem a cliché idea, and poor execution can turn a great idea disappointing. ((5): Severe weakness. I was never able to relate to the narrator, and Dash felt like a cardboard cutout. The plot felt extremely arbitrary, though there were a few authentic moments. Without building those connections, the core tragedy falls completely apart.)
Make sense?
Normalization
Again, HORSE is NOT a judgment of quality, and should not be used that way. (It can easily be combined with a system that DOES render such judgments, like a 1-to-5-star rating.) Because HORSE is normalized at 100 points per story, the goal is not to "grade" the story, but to give its author an at-a-glance view of what areas are most crucial to address with editing/rewriting. Don't think that your scores make you "better" or "worse" than other stories with different HORSE scores.
The only thing a HORSE score can be usefully compared with is the SAME STORY'S scores in other categories. If you score 'Ritin-50, it doesn't necessarily mean you play with language to put Shakespeare to shame; it just means that a proofreader should be the last thing on your mind when you're revising, because there are lower-scoring categories that need more urgent attention. If you score Execution-50, it doesn't necessarily mean you wrote a sadfic that could bring tears to the eyes of grinches; it just means that the core of the story does what it needs to do, and your writing is being dragged down by factors in the low-scoring categories. Similarly, Originality-0 isn't an accusation that you have no creativity, it just means that the reviewer looked at this particular story and thought the writing talents in your high-scoring categories were being paraded down an exceptionally well-trodden road.
The best story in the universe would have a 20 in every category, because it would all be equally brilliant. A story that made my eyes weep blood would have a 20 in every category, because it would all be equally irredeemable. The things that we mortal authors write will fall in between — and they'll be good and bad in different ways.
If you have limited editing time (and let's face it, all of us do), then focus on the areas with lowest scores first.
See It In Practice
I gave HORSE ratings for 93 stories (!!) over in the feedback thread for the Writeoff Association's January 2015 competition. This comment has links to all seven of the batches in which I deployed the reviews and rankings.
To Use It Yourself
If you want to give HORSE ratings to stories you read, then provide the author an explanatory link to this post. Then you can simply copy and paste the following codes, changing the numbers or the lengths of the bars:
[b][color=#ea80b0]H-20[/color] [color=#6aaadd]O-20[/color] [color=#e97135]R-20[/color] [color=#5e51a3]S-20[/color] [color=#e6b91f]E-20[/color][/b]
…will give you
H-20 O-20 R-20 S-20 E-20
[b][color=#ea80b0]H[/color][color=#6aaadd]O[/color][color=#e97135]R[/color][color=#5e51a3]S[/color][color=#e6b91f]E[/color]:[/b] [b][color=#ea80b0]▉▉▉▉[/color][color=#6aaadd]▉▉▉▉[/color][color=#e97135]▉▉▉▉[/color][color=#5e51a3]▉▉▉▉[/color][color=#e6b91f]▉▉▉▉[/color][/b]
…will give you
HORSE: ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉
[b][color=#ea80b0]HHHH[/color][color=#6aaadd]OOOO[/color][color=#e97135]RRRR[/color][color=#5e51a3]SSSS[/color][color=#e6b91f]EEEE[/color][/b]
…will give you
HHHHOOOORRRRSSSSEEEE
Along with my HORSE score, for the writeoff stories that I'm judging in the HORSE test run, I'm assigning each story into one of six tiers. These loosely track the overall vote I'm giving a fic, but have some overlap with each other:
Misaimed (M) — I, horizon, will fundamentally never appreciate this story, for reasons that have nothing to do with the quality of your prose. If you care about that, I recommend a ground-up rewrite, or scrapping it and moving on to a different story concept. More likely, though, I'm not meant to be your target audience.
Needs Work (NW) — I feel this has substantial flaws to iron out, but all writeoff stories are basically first drafts, and with editing, every story has the potential for greatness. Keep it up!
Flawed but Fun (FF) — I saw substantial problems here, and you probably should brace for a poor score in the overall competition, but this got the most important thing right: it entertained me. Thank you!
Almost There (AT) — I can already see the greatness in this one, but there are some specific things holding it back, and they should be pretty easy to address with editing. I look forward to seeing the result!
Solid Strong (S) — I felt this did what it set out to do. Good job! It didn't grab me the way my top tier did, but there might be style preferences or minor editing issues at play.
Top Contender (TC) — This stood out from the pack. Both enjoyable and well-constructed!
(Edit 6/2016: Changed name of S tier.)
Dear Princess Celestia, today horizon learned how to market an idea.
Best
I don't know how I missed your original HITEC post (actually, maybe I was at BUCK at the time), but gosh darn this excites me in a way that only colored numbers and mathematical models can. I don't have any criticisms yet, I'm still running through the possibilities in my head. But I like this. Except for the excruciatingly forced acronym. ^^
also how you do escape square bracket markup in text even go to look more like?
2702002 Particularly because it's Applejack-endorsed.
2702015
nay, sir, it is an excruciatingly HORSE acronym :V
2702032
APPLEJACK: "Ah learned the three Rs at school: reading an' ritin'."
I think the biggest problem with this system is that it only seems to be useful to the author.
For a reviewer, they want a system that can express the quality of the story at a glance to an audience, to sum up whether or not the story itself is worth reading. A yea or nay to potential readers.
What this accomplishes is a system that is incredibly useful to prereaders and editors and fic-rejectors. If you can't find the appeal to it, though, for general reviewers - the people who will be seen and heard by the most people who might express interest in the system - then you're going to be stuck at the gate.
Which is a shame, because it's genuinely an interesting system.
Hook/Story/Characterization/Grammar. Hm. No, my normal checklist doesn't have a good acronym.
(Note to self: HORSE started here)
2702021
There's no canonical [nocode] support AFAIK, but the bracket escapes take ugly advantage of a… shall we say feature… in FIMFic's BBCode processing: it does single-pass parsing.
So if you want
Boldtext! [1]
You have this source code:
[b]Boldtext![/b] [2]
And if you want to display that, then your goal is simply to break the tags so they no longer look like BBCode when the parsing engine passes through. The simplest way is to insert a blank pair of tags within each tag (the keyboard shortcut Control+i will drop an Italic set at your cursor):
[[i][/i]b]Boldtext![[i][/i]/b] [3]
The parser will turn the italic tags into HTML formatting, but there's no enclosed text to format, so the end result shown on screen is [2] above.
Further recursion is left as an exercise to the reader.
Hmmm. I like this. But I'm torn... The boxes are really lovely looking, but I also find them much harder to quickly parse, compared to the 10/15/20/30/25 numbers.
2702105 This is a feedback system, not a review system. Feedback is exclusively about assisting the author.
2702433 - cool, thanks! Although breaking markup in that way discomforts me on some almost-subconscious level, so I probably won't do it. ^^
2702105 2702708
The author-centrism is, as noted, an intentional part of the design. It's easy enough to tell readers what's worth taking a look at — check out, for example, 2702041's reviews and his various recommendation levels. The flip side of that is that reviews ought to be useful to authors, too, because telling them the best ways to improve the story (or their next one) leads to authors more excited for feedback, and ultimately a higher quality of fanfic.
> If you can't find the appeal to it, though, for general reviewers - the people who will be seen and heard by the most people who might express interest in the system - then you're going to be stuck at the gate.
Well, then, eh. If it's a niche tool, then it's a niche tool. I'm a niche kind of guy, and the world will always have a place for niche tools.
Even if it's never widely used, getting folks to think is a noble goal, and I've got enough fellow reviewers watching me that it might bear unexpected fruit down the road.
I like it. As you say, it is a niche tool, but as a writer and reviewer, that's the niche I need tools for. If I want to know if a story is GOOD or not, I can just say "Read that" or "Don't read that" and it covers 90% of the cases. The details of how to improve are what I want to learn (or impart) most often though.
As I'm about to start reworking the Iron Author contest codebase for online use, I wanted to integrate something like this into it. People should get judgement AND feedback on stories in a contest I think.
I do have one complaint though: Your color choices! While I appriciate the thought to use the non-princess-ponies as stand ins, the way you've lined it out isn't exactly intuitive. Sure, it makes sense with the stories you tell here, but it's not an easy device to remember them by. Plus, Structure as Rarity as Purple just breaks my brain. If it's purple, it should be Twilight at the very least. And Rarity (in fashion) should be about originality. And Dash should, of course, be most concerned with Execution, because she's perfect in her stunts. Fluttershy is the one worries about first impressions, because Pinkie is the one...
See? See what you did there, you broke my poor, defenseless brain!
In all seriousness though, I really do have trouble remembering which color is supposed to go with which idea, and my dyslexic tendencies are causing my brain to fail halfway through trying to decipher the color-bar scoring. I really think you should make it simpler, and just go with color names. That is, Structure should be Silver, Salmon, or Sienna! Then I could use the mnemonic for the meaning AND the color.
2704046
Thanks for the kind words and feedback. (If you want to use this yourself, please go right ahead! [1])
To be honest, I had to make your intuition cry for one big reason: the original ordering left several similar colors adjacent to each other, and the current version flips between light contrast and dark contrast so that you can pick out different sections at an immediate glance. (It should be colorblindness-safe, too, but I'll have to run it past my red-green colorblind friend to double-check.) That's also why I ended up using Rarity-hair purple but not Twilight-coat lavender; I couldn't tell their colors apart and purple worked better with the others. The arbitrary little narrations were less an attempt to provide a mnemonic and more a way to offer a flimsy justification for the ponycolors.
You shouldn't have to remember the colors, though -- just copy and paste the templates.
Personally, I remember them by position, but I can see how dyslexia would make that more difficult.
If you can come up with colors that work for the scale and display well together and have the right mnemonic, I'd be curious to hear 'em!
--
[1] To be technical about it, consider the idea and implementation shared for public reuse under a Creative Commons BY-NC-SA license [2]; I'll go fish up the links and a badge at some point when I'm not being sick and/or lazy. If you want to use this for Everfree and they require something more substantial than that, drop me an email and I can waive the NC for them or do a separate (free) license.
[2] This means, among other things, that as long as you're willing to similarly share your changes, you can tinker with the system all you like.
(EDITED TO ADD: Relicensed under BY-SA in July 2015.)
2705958 I don't mean remember the color names themselves, but rather, when I look at the color bar, and see a lot of one color then a little of the second or third, I think, "hmm, the story was poor on that trait, the orange one, right, originality!" which is wrong, of course, so then I have to spell out "H-O-R-S-E as I count color changes in the bar. Yes, I know I should be able to count to five with less effort, but I still have to sing the ABC song in my head if I need to know what letter comes after G as well.
Bottom line, if I read it slowly left to right, it's doable, but since it's visual, my brain wants to treat it like a chart, where you can pick any element and "index" it on color, not it's position in a sequence. E.g. I look a histogram of 30 categories, and I see a really tall bar for the 20th one... I don't have to read the first 19 in order to know what that one is.
All that said, you asked for a better display. Hmm... How about this:
HHHHHHOORRRRRSSSEEEEEE
Works if you're colorblind OR dyslexic, copies into plain-text medium without information loss, the "legend" is encoded directly into the display for higher information density (so you don't need the prefixed key/title), it still provides visual/graph-like structure, and (provided the story isn't too poorly executed) your brain gets to squee "HORSEEEE!" in an overly saccharine internal monologue!
I've got that badge on the bottom corner of xepher.net. I try to release most anything I work on (at least for fun) under a CC license, so I'm quite familiar with it. That said, thanks for the permission, I was going to talk to you about it privately later if I wanted to go that way. I'm not sure yet if it'll be good for the contest or not though. Mostly because I worry less technical people will get confused about how it doesn't rank stories against each other, especially in a contest system.
2706246 horizon, you asked for feedback on the following suggestion:
I'll be honest, I spent more time pretending to scream the word "horse" at the top of my lungs than I did reading the reviews. Not that that wasn't a fun, enjoyable use of my time, but I still feel like it might not have been the intention.
It does look a little better, though.
I find the solid bars a bit easier to process at a glance given the letters' different widths, but I'm not color-blind, and the letter model is much funnier.
"Not bad. I give it a HHOOOOORRRRSSSSSEEEE."
"... Is that good?"
"Sorry, I had something caught in my throat. I meant HHHOOOORRRRRSSSSEEEE."
I have given further thought to the ideas in this and your HITEC system, and forked many of the elements into a system of my own that I am calling TAILS. Consider it another step in the discussion. Should you take the time to give it a look, I'd be very interested to know what you make of it...
TAILS (beta) blog post