3957543

The problem is that, because of this divide, I feel like there is a very real chance that stories aren't getting judged fairly. I mean, I get that everyone will determine a story's score differently, but that's not the same thing as everyone evaluating scores based on completely different and incomparable structures.

Perhaps I'm just being dense, but this seems like the start of a very good argument for using exactly the system the write-off uses, and focusing on voters' unique information like I mentioned above.

You're right. People are evaluating stories based on completely different and incomparable structures. To my understanding, that's how just about every creative work, in any domain, ever, gets evaluated. That's how a lot of non-creative works get evaluated. That's sort of the core nature of forming judgments about things you experience.

My personal view is that the fairest evaluations are those with the most fidelity to reality, and because of that, evaulations based on any sort of fixed rubric make very little sense to me. The reading public doesn't have anything like a fixed rubric for judging stories—all they have are a lot of (often conflicting) opinions and feelings. To me, votes should be flexible in the way readers' own judgments are flexible, to best approximate reality.

Maybe you're arguing for some sort of objective standards against which writing should be evaluated? I've never been able to wrap my head around that kind of idea; it feels too anti-creative to me, to try to systematize how people report on what's at its core a deeply personal experience. But maybe that's not what you're arguing for? I've read all your posts here, and I can't really figure it out. You seem to want story feedback in a form I can't wrap my head around. I'm really not sure how to even participate in this discussion.

3957747

I entirely agree that standardized scoring, while it would certainly helpful to find as close to an objective evaluation as possible, is entirely impossible for a medium such as writing. My problem isn't that people are using vastly different structures to generate their scores. My problem is both that the scores themselves aren't at all indicative of the means used to produce them and that all these scores are being dumped into one lump metric that can be interpreted in many different ways (if it's even possible for them to be properly interpreted at all).

At the moment, it just seems like the system could be seen in one of two ways.

[A] People need to go to the reviews in order to understand why they got the votes they did; or
People go to the reviews and disregard the votes entirely.

In the case of A, the reviews are serving the votes, which my impression of the goal of this event would argue that it should be the other way around. In the case of B, the votes are completely meaningless, in which case there's no reason to even have them.

3957771

My problem is both that the scores themselves aren't at all indicative of the means used to produce them and that all these scores are being dumped into one lump metric that can be interpreted in many different ways (if it's even possible for them to be properly interpreted at all).

Okay, could I ask you to unpack that first part, please?

The second part strikes me as standard CLT / law of large numbers, and means that you're never going to be able to get a summary that really means anything other than "the average opinion of readers", but I'm fine with that. Then again, I'm in the same camp as folks who are only looking for feedback from the reviews, and in terms of rank-ordering on stories, so I can get some feel for which stories are being grouped around the same level as mine.

The first part is the part I don't think I understand. I'll refrain from suggesting interpretations again, because I honestly don't think I know what you mean by it.

3957793

Okay, could I ask you to unpack that first part, please?

What I mean is that there are a thousand and one different ways that different people are using to determine how they score any given story. Maybe they use a rigid structure like Quill Scratch. Maybe they weigh it on something similar to a HORSE scale like horizon. Some people assign specific scores for specific impressions of a story like bookplayer. Maybe they grade on a curve. Maybe they grade based on a gut reaction. Or maybe they use any number of other means that I can't even conceive of. Any of these are perfectly legitimate ways to come to a score, and I'm not arguing that.

However, when it's all said and done, all the author is left with is the result of all those different structures - a number based on a scale, without any means of reading into the structure or thought processes of the people who gave it. And with so many different ways to evaluate the score, this is a problem.

Say if, for example, one person graded a story using a rigid categorized approach, concluded that the flow and characterization were decent but lacking in narrative and technical skill, and ended up giving it a 6. Then another person went more with how they compared it with other entries of a similar theme and thought other stories pulled it off better, and gave it a 3. A third person thought the use of a particular icon was original and creative, so they deemed it worthy of an 8. All of these scores are based on a different systems, and within those systems they are as "correct" as anyone could realistically argue them to be. But in the end, all the author sees is the 5.6 average of the three numbers, and thus don't get any of the meanings or insights of the individual scores. Even if they could see the 8, 6, and 3, they would still have no way to know how they were supposed to be interpreted. They would instead default to their own understanding, which could lead them to a conclusion about their story that isn't at all what the voters had in mind.

This is why I argue that the voting system should be tweaked to give more meaningful feedback. But then others argue that voting is meaningless for feedback purposes anyway, in which case I argue that maybe they should then be removed completely.

The second part strikes me as standard CLT / law of large numbers, and means that you're never going to be able to get a summary that really means anything other than "the average opinion of readers", but I'm fine with that.

If the goal of this event was competition, then I'd be fine with that too. But I was under the impression that the focus of this event was for more specific criticism and analysis rather than just being assigned a position in a single file line. If I'm incorrect in that assumption, could someone correct me there?

Then again, I'm in the same camp as folks who are only looking for feedback from the reviews, and in terms of rank-ordering on stories, so I can get some feel for which stories are being grouped around the same level as mine.

This ties into my original concern with the vote weighing, though. I too am part of the camp that would like to see where my things rank in comparison to others, but now that I know there are people who are specifically tweaking their votes based on arbitrary metrics, how can I know that those ranks are accurate? Suddenly I find myself second-guessing the votes I received, and wondering if I don't actually end up higher or lower than what I get. If I can't trust the system that is giving me my scores, what is the point of me even taking part in this write-off?

3957879

If I can't trust the system that is giving me my scores, what is the point of me even taking part in this write-off?

.... The reviews. And the motivation to get a story done.

Honestly I do not understand why you're placing so much value on the vote. It's an ego boost if you do good, and if you don't it's not the end of the world. It isn't the most accurate thing, but nobody cares because we're all here for the reviews anyway.

And before someone asks "then why not remove it?", it's because it does not impact the reviewing. Maybe the reviewing impacts scoring, but that's not the primary objective of the writeoff.

I honestly like it that the write off has a low-competitive vibe. I'm not saying this is everyone's cup of tea, but to be frank there are plenty of other places you go can go if you're so determined to get an absolute/"objective" assessment of your story.

3957915

I'm not sure why everyone seems to think I'm here to get a strictly objective analysis of my writing. I am here to get any analysis I can, and while I can honestly say that I appreciate the reviews side of the write-off, I was also looking forward to the votes as well. Not because I want to win a competition, but because it's another way to get useful feedback if handled well.

And having 50 people give me 50 different scores for 50 different reasons, and being unable to get any insight into any of those reasons (other than maybe five of them through the reviews if I'm lucky), along with learning about practices that artificially skew the vote averages, is not what I would consider being "handled well".

See, this is why I am saying there should be voting guidelines. Not anything hardcore like 'Vote based on HORSE' or such, but something simple like 'Your votes should be relative to other entries in this contest' or 'Vote based on how you feel about the story as a piece of fiction as a whole' or whatever, and similarly 'Try to vote on a bell curve' or 'Try to normalize votes from 1-10' or whatever.

Because we don't have a voting population high enough to cancel out variance. The last 3 contests had 22, 18, and 25 votesets, and those aren't guaranteed to be 100% complete since you only need vote on half. This one has 16 votesets in currently - so yes, 2 or 3 people voting like TD, even if they aren't collaborating at all, can have a massive impact on the end results if they end up overlapping preferences in a significant way.

I'm not advocating a rigid set of guidelines, just a loose framework on general scoring. Basically I think there are 2 metrics that should be agreed upon and established :
1. Should stories be voted on relative to the current contest, or relative to their merits as a whole? The rule amongst regulars is generally current contest, but I see a lot of new people - and I know I did it that way when I was new - where I voted relative to their merits as a whole. And honestly, there's no reason that these can't co-exist, but if so it should be noted you're free to vote either way.
2. How should votes be distributed? This is the one I do feel we really do need some established framework on because the voting group is generally 25 or fewer, and where I think it should either be 'vote on a bell curve, aim to average a hugbox of around 5', or 'In a story with 33 entries, have 3 of each score, with again a hugbox of 5'.

Beyond that, do whatever; vote upon if you like it, or if it has literary merit in your eyes, or form or function or characterization. Because I really really do not enjoy feeling pulled between 'maximize my own voting return via game theory so that other people aren't twice as potent as me' and 'Vote with what feels fun, enjoyable, and right'. And right now I am and continue to feel pulled between those poles and I do not like that.

I feel somehow compelled to chime in on this thread, but my reaction to doing so is along the lines of "UUUUUUGGGHHHHH" and I haven't read much after Bradel's second (very well thought-out, I should mention) post.

I don't give two shits how people vote, so long as there aren't cabals forming. I think a regimented voting system is unenforceable and a bad idea. Bradel definitely has it right when he says that large numbers of voters reduce the power of any single voter's numbers. I think the effect that TD or bookplayer have on the contest is overall negligible.

Of course, if you want to get the most out of your vote, give out no scores but 0, 3, 7 and 10.

Here's how I do it, since no one asked:

-After reading the story, I think, "How much did I like it? How much do flaws in the story affect this?"

-I assign a value to this based on the old EQD star rating system: 5 means it was excellent, 4 means I really liked it, 3 I liked it, 2 I didn't like it and 1, it was bad.

-I then double this number (.5s are allowable, too) and subtract one, giving me a score from 1-9. 10s I reserve for 6-star fics or, if there aren't any, my favorite from the event. 0s I give only to stories that both lack merit and are poorly put together, or that attack the reader as some have in the past.

-I then normalize the scores on as best a bell curve as I can make. I didn't do it this time because there was hardly what I would call a reasonable number of entries. :|

-Bob's your uncle.

Obviously, everyone should be more like me and then the world would be a happier place. :V I don't know what the rest of this thread is about. All I know is that Closing Time is a conspiracy, a group of people are "all in" on it, and I'm mad because I'm not one of them. :(

3956604

PresentPerfect does too, but he hates all my stories, so I'm going to ignore the fact that he exists here. Hi, PP!

I'm laughing because you've already gotten yours, buster. :V

3957951
It is for all those reasons that the voting results aren't taking very seriously.

Honestly, what can we do instead? If use a defined scale for each number score, as in "0 = terrible; 10 = amazing" this only makes it easier to cheat the system by impacting averages via throwing 0's at the stories you think will do better than yours.

The only way to make votes a significantly meaningful form of feedback is if the voters were impartial, which cannot feasibly happen because that would mean voters would not be able to submit their own entries.

Let Drew Carey be your spirit animal...

No but really, I think both sides are right, in their own way. A single "2", in a vacuum, is meaningless. When you get 20 scores together, the meaning of the individual numbers gets smooshed into an average. And while a 5.84 means nothing again in a vacuum, I can look at someone else's fic which got a 6.42, and gain relative understanding—nothing more, nothing less. For everything else, there's MasterCard detailed comments.

3958258
What's great about this post is, with the author guessing now codified, this has really become Whose Fic Is It, Anyway?

Since then, I've tried to normalize things so each # from 1-10 has an equal number of fics in it. (For that matter, I don't know why we have 0s on the scale; we may as well make it go to Eleven at that point, because 12 is a nicer number than 11). So, stories are voted relative to one another; if every entry were the most amazing thing possible I would consider upending that but generally it works out okay, though it does mean that stories that are 4s and 5s in the general populace are getting 1s and 2s here. But that's fine, that's just reflecting strength of competition.

One advantage of having a 0-10 scale is it makes 5 average; otherwise, the average would be 5.5. With 5 as the average, there are an equal number of numbers above 5 (6-10) as below it (0-4), and 5 being the average is more intuitive than 6.

It also is a convenient divisor of a percentile scale, as 0-100% divided by 10 gives a 0-10 scale.

For me, I really really would love to see a consistent 'Voting should be done in this style' metric. For my first writeoff I voted based on how I felt the stories rated relative to fiction as a whole, meaning my average was in the 6s somwhere, because getting a 0/1/2 would be really really hard since those would be the shit that just barely passes moderation on Fimfic.

I don't know if making voting more complicated is desirable; while in principle rules for it would be nice, in practice I'm not sure that people would adhere to them very well anyway because, quite frankly, training people how to judge things properly on a scale is a lot of work.

The issue I see is that we're not consistent in this application. Some of us are grading where each point gets an equal number of entries; some grade on a bell curve; TD grades on curve where 0 is the baseline and the most commonly used score. And yea, I find this frustrating if the voting system is supposed to have some kind of meaning because it does invite gaming it in various ways, and a 10 from him remains a much more potent 10 because most other people are getting like 3 or below.

My average score is roughly 4 (my hugbox is 3.80; it fell slightly from 4 this time). 0 isn't always my most common score, either; it wasn't my most common score for any of the short story competitions, IIRC. The minific competitions just seem a bit prone to producing stories that end up not going anywhere.

3956440
Have you ever read The New Yorker? The stories there are probably a good example of "genius, but boring". Well, some of them anyway; sometimes they let something which is actually good slip through.

Next post will be devoted to Bradel.

3957994

I agree with this. Keep reading if you care to know why I think this.

The Writeoff — what is it?
Is it a competition to win a gold medal?
Is it a chance to have fun with friends?
Is it an easy way to get reviews on your next big story idea?
Is it a way to prank people and screw with all of the above?
Is it none/all/mix of these things?
Is it something else altogether?

Only you can decide what it means to you. Writing a list of rules and trying to enforce said rules isn't going to codify any of that. It may change people's opinions of the competition, and probably not for the better.

The same goes for voting.

People can and do say pretty much whatever they want. They can mark their vote sheet with random bubbles for all I know or care. They can say they voted one way and not actually do so. It's just one vote. The only problem for the competition comes into play when people work outside of the voting sheet to "hack" the system. This includes directly hacking the site, multiple accounts, cabals/group voting, etc... This does NOT mean the influence of reviews and feedback constitutes a hack. Even if this can influence the score someone gives, it does not imply any unfairness or impropriety. Everyone controls their own vote. I can't vote for you or change your vote in any direct way. The closest I can come is to tell you which story is mine and let author bias intrude on your choice, and in the process I disqualify myself anyway.

As far as making a standard for how everyone is expected to vote, I can't support that. This is not a contest. This is more of a game. If it was a contest, you could argue that there should be a set panel of judges that take the same criteria in mind when handing down their standardized votes. I say this is a game, because there are no voting rules. If you want the gold medal, you have to write a story that more-or-less sways the varying methods of voting from a large number of people in order to put your fic on top. It has nothing to do with how amazingly awesome your story was in the general world of literature, or even in the realm of horsewords. It only means you got a large enough consensus of the voters here to rate your story higher on a subjective scale than all the other fics in this particular round. You can't even compare fics across events. That's all it means.

If you really care about min/maxing the impact of your votes, feel free to debate that all you want. I'm perfectly happy with any system or non-system people use to vote on the stories. If they want to be a jerk and rate my story a 0 for no good reason at all, that's their call. I really have no way of knowing unless they actually post their scores and I believe what they posted is true. What I can do is take that feedback along with whatever score I receive relative to the other stories in the round and do my best to try and make my fic better given what I have learned. If I don't care about making it better, I don't have to. If I want to just write the story for my own gratification, I can do that. It's kind of assumed that people here care just a little bit about the response others give to their work. If you don't, I have to ask, "why are you here?" But that goes back to my first point, 'What is the Writeoff?' Only you can answer that for you, and only you can vote on what you think the scores should be. I'll do the same, and I'll/you'll/we'll all have a fun/not fun time together writing/reading/reviewing/dicussing/voting/guessing horsewords for the enjoyment/detriment of ourselves/others.

(Just to be clear, I interpret that last line like this: We'll all have a fun time together writing, reading, reviewing, discussing, voting, and guessing at horsewords for the enjoyment of everypony!)

P.S. I used the to denote my sarcasm this time. I need to do a better job at that seeing as none of you really know me that well.

There will never be a codified standard or rubric for votes. Votes mean whatever the voter wants them to. It's most optimal to vote on a relative scale, and you should assume most people do, but you don't have to vote that way. (Just know that you're voting suboptimally.)

I have some qualms with changing your vote based on others' votes. Namely, your vote must remain internally consistent. A 5 must be, by whatever metric you're using, better than a 4 and worse than a 6. If you believe a story to be equal to another 7 but give it a 6 because others overrate it, then your record is not consistent with your criteria. Put simply, it's dishonest. You are only supposed to judge the story itself.

The write-off is a competition. There is some degree of meritocracy to the rankings. This allows participants to enter feeling that they need to give it their best. Nonetheless, the scores should be taken with a grain of salt because they are subjective and there is much luck involved. They are a tool in the kit of learning that the write-off provides.

The final score your story gets is indicative of one thing: how well your story was received by write-off voters. If you want context of where your number came from and how to improve your work, read the reviews. Don't just read reviews of your entry. Read all the entries, all the reviews, and your own reviews. Figure out what this audience does and doesn't like, what does and doesn't work for them, and how this lines up with your own impressions. There is an absurd wealth of information to learn from in these events if you use it properly. Spoonfeeding with categorical or contextual votes will only overcomplicate the process without purpose.

3956441

Does this all say anything about how people should vote? I think yes, sort of. And I think, of all people, Titanium Dragon has this closest to optimal.

I'm like quantum physics - the only thing I have going for me is being right, because Celestia knows it isn't anything else.

a lot of 0's and 10's

I don't give THAT many 10s. I just seem to because a lot of folks seem to hand out only 1-2 of them. Handing out 5 seems like a lot if you're creating a bell curve; I don't create a bell curve (my distribution is closer to a power curve if anything, but it is very flat on the tail) and really, it isn't always that way; again, on the short story competition, my mode score was 4. The reason that there were a lot of 0s is because it is much easier to miss the point in 750 words accidentally than it is to write 2,000+ words and not really know where your story is going. With 750 words, you have like, one scene, maybe two, and it is possible to make that be the wrong scene and just not end up going anywhere, or having to cut so much that your story doesn't make sense or whatever.

3956604
To clarify:

Here is my distribution from this competition, from my notes:

10: 5 (Safe, Dignity, 110%, To the Sun, The Sea Pony)
9: 4 (Faults, Protracted Thoughts, Leap of Faith, Method Acting)
8: 4 (Falling in Love, Open Book, A Spoonful of Sugar, Small Horse Poop Tale)
7: 4 (Verdict, Fetch, In Service of the Princess of Friendship, Anything and Everything)
6: 6 (Strength, Magic is Arbitrary, All in the Presentation, A Nugget of Wisdom, Apple Bloom Tempts Fate, Big Trouble)
5: 7
4: 12
3: 14
2: 9
1: 10
0: 18

I will note that originally, I had no 9s at all (I had the 10s standing above what is now at 9s, with them at 8s, the 8s at the 7s, and what are now 7s mixed in with the 6s) but I decided that was a bit dumb, though it did give a little less advantage to the 10s, which is unfortunate. I may end up going back and changing it back, but... eh, I dunno, feels weird to have a big gap up at the top.

So why were there no 9s? Originally, I put The Sea Pony at 9; however, after additional consideration, I decided that it was too good to be a 9 and deserved to be a 10, as retrospectively it ended up being one of the very strongest in the competition. I went through my rationale for these top stories in the other thread, but I really felt like the 10s I gave out were all very good, and the 8s I gave out were to good stories, but they weren't as good... I felt like there was a gap there. So there was one.

Still, you can see what more or less happened with my scores: the top scores ended up pretty spread out, with the 0-5 being more clustered, letting me give more differentiation between a story like A Nugget of Wisdom and Safe.

I still feel terribly awkward giving Small Horse Poop Tale an 8. But it just strikes me as too ridiculous to rate lower than its fellows, and bumping them all up... well, anyway, normally 8-10 are my "top end" stories, 5-7 are the "goods", 3-4 are mediocre, 1-2 are weak, and 0s are stories that end up really falling down.

Point is that it has a lot over on the low-end, with a fairly thick tail for the outlier "good stories". Ordinarily, this results in the worst of the "top end" stories getting twice as many points out of me as the top of the mediocre stories.

I don't have a problem giving out more than one 10 because I think seeing any of the stories I gave 10s to win would be acceptable to me, and I feel like they're probably the best.

But if I read a reviewer discussing a story and they've noticed things about it that totally went over my head, and if I think those things are important and I was dumb for missing them, then of course I'm going to let that impact my vote.

I agree.

I think it still feels wrong to a lot of people, and one of the reasons (which I think bookplayer has tried to clear up, to middling success) is that she wasn't initially as clear as she could have been that she was talking about voting more negatively on stories she already actively disliked, as opposed to trying to game the system so some good stories couldn't win. Even if a story is good in many readers' minds, it may not be good in everyone's mind, and it makes sense to give low votes to stories you don't think are good.

I was understanding her to really be saying more "This is how I was supposed to vote, but didn't because I usually feel bad about giving out 0s" rather than "I am trying to screw this story's score over", as noted by her rating scale above.

3958894

Then at least please note 'People can use whatever metric they want as their own personal guidelines for voting' in the rules so that is front and center. Because right now, this discussion has come up at least twice since I started because there's a good deal of confusion in how voting is supposed to work.

3958974

I was understanding her to really be saying more "This is how I was supposed to vote, but didn't because I usually feel bad about giving out 0s" rather than "I am trying to screw this story's score over", as noted by her rating scale above.

To clarify, I'm saying the same thing everywhere.

To put it bluntly, to be absolutely clear:

Some fics are by new writers who were trying to tell an interesting story and failed. I don't think these fics need zeros to express that, a 3 will put my vote below average in almost any round, but they'll be softer about it.

Some fics are by writers who apparently know what they're doing in terms of writing, and they decided that their story was too cool for an understandable plot. These writers are in the big leagues, they don't need a safety net, so for them I'll play the game.

Of course, now with word from Roger I'm considering give a zero to any fic I found boring. This sucks, because it will throw off my scale relative to other new writer problems like formatting-- I'd hate to give a fic with no new paragraphs for dialogue a three and give a technically okay but boring fic a zero -- but if people don't want me to take into account information I probably missed because I was busy being bored, I'll have to assume that most competent but boring stories are written by people who know how to tell a story and choose not to. We all know this isn't true, which is what bothers me about it, but... what else can I do?

3959049

what else can I do?

You can forget about what other people think about voting and do whatever you feel works best for you. I'm pretty sure the general feeling here is that as long as you stay true to you, that's what matters most.

I don't let other people tell me how to vote. I make up my own mind and consider input I gather from the reviews after I put down my initial thoughts. Given the whole week to mull things over, I might make a couple minor adjustments. This is never more than a point or two. Going back and knocking a top story down to a zero because I don't want them to win is not the right thing for me to do. Others are free to make their own choice. Even if I don't think it's right, I can't force them to follow my rules.

If I went in with the mindset that I'm giving a 0 to things that bored me or were too kitschy for their own good, then I'm still being consistent, and that's okay with me. That's not my rationale for voting, as I explained in the other thread, but I have no problem with others taking that view.

3956689
Story is a major part of how good a piece of fiction writing is. If you don't have anything to say, if your story lacks a climax or a purpose, if your story doesn't do anything interesting, then it doesn't matter whether or not your grammar or spelling are spot-on - you've written the wrong thing on the page.

A really beautifully written but utterly pointless story might win some points from me, but it still isn't likely to win a top spot from me unless it does something very clever indeed.

That doesn't mean I don't care about technical writing - I do - but in the end, story is very important to my enjoyment of a piece. I do read these to be entertained, after all.

3956741
Shipping is always dicey, but I do tend to agree. Then again, I'm happy to accept any ship... as long as the story convinces me of it. Likewise, if a shipfic is poorly written, even if I ship it like FedEx, I'll still downvote it.

Needless to say, this means that no one wants me to read their shipfics anymore. :fluttershycry:

I'm making myself read what is supposedly the best FlutterDash shipfic (or one of them, anyway) for my next Read It Later, though. Hopefully I'll like it.

But it is very hard to establish a relationship in 750 words. Or, more accurately, to get people to buy it - you don't actually have to justify it, you just have to trick the audience into not questioning it while they're reading the story. This is very hard for any sort of main character.

3957060
I've gotten not that reaction, but other negative reactions, yes. Like my poor AppleDash story that I still need to cut apart and rewrite because it is a mess currently.

3957098
You can't always distinguish between the two, though, unless it is something you specifically know you dislike (a ship or whatever you dislike, if you're the type).

Then again, you still don't like it.

3957195
That's because Pascoite was looking at a limited set of reviewers and assuming they were representative. Even my story made someone's top 5 that round, and Through Glass came in 16th (i.e. dead in the middle of the 31 entries).

It is also worth remembering that something on the order of half of the stories in Behind Closed Doors made at least one top 5 stories list, so it isn't necessarily very meaningful.

3957356

You're still not getting my point. These scores make no sense outside of the event they were written in. They're not supposed to. Getting a 9.7 doesn't mean a story is amazing. All it means is that relative to the other stories in the event it was submitted into, it was generally well-received. You cannot compare this score to a fic that got a 8.2 in a different event. The point of scoring isn't to determine a fic's quality. It's to compare how well-received it was in relation to other fics. It's been mentioned in other threads and by Roger that the 0-10 scale isn't absolute value.

This is worth keeping in mind; if we actually got a 9.7, that would probably indicate either that the story was totally amazing or that everyone else's entries weren't nearly as strong, or some combination of the two.

If you look at, say, Title Drop, you'll see that there was a big gap between All the Mortal Remains and Collaborators, and then another big gap between Intern and Celestiology, and then ANOTHER big gap between Celestiology and everything else. These are indications that these stories were seen as much better than other stories. Just Over the Horizon showed a similar distribution, with The Storm winning by half a point, and then the separation between 4th and 5th being more than a half point - and first place had nearly a full point over 4th place already.

On the other hand, Famous Last Words was very tightly packed, score-wise - First and fourth place were separated by only .22 pointsand first and 13th place were only a point apart. Compare that to Title Drop, where you see that first and fourth place were more than a point apart, and you can see that Famous Last Words was a much tighter competition. There were a ton of fairly equal stories submitted that competition (and they were all quite good; honestly, I think that was the most impressive showing so far in terms of top stories, at least for me). There is Magic in Everything had a very low high score (only 7.11), and the top nine stories are all within a point of each other.

For a really extreme example, History Repeats had The 18th Brewmare get a score of 9.25, Ἐλπίς get a 7.62, Naval Gazing get a 6.56, and Memories of a Star get 5.38 (with it and the next two all being within 0.2 points of each other). There was very clear agreement there over who deserved to win, but that doesn't necessarily mean that 18th Brewmare was the best writeoff story ever, nor does that 7.62 on Ἐλπίς necessarily mean it was better than every single story in There Is Magic in Everything (though both are arguments you could probably win; that round had a couple of ringers).

3957358
I must have missed that comment.

3959231

That's because Pascoite was looking at a limited set of reviewers and assuming they were representative.

That's not really accurate, and it's dismissive of a lot of what I said. The reviewers I was looking at were people like Chris, horizon, and PresentPerfect, who have really earned their reputations as good reviewers, and I noted that such people used to be very indicative of overall finish order until we got the huge influx of participants, i.e., the new people don't value what the good reviewers do.

And you seem to have completely missed my point about the difference between a story contest and a writing contest. I'll give a good score to a story that I didn't like when I can isolate my distaste of it to personal preference. All the things you noted are justifiable reasons from an objective standpoint to deduct score, which is not at all what I'm talking about.

3959859
Well, yeah, but there were only 9 votes on History Repeats, which means that they would have been a third of the votes on it, and there were only 7 stories there. They would, by necessity, be a much better indicator of the top 5, both because there were far fewer votes (and thus they made up a larger fraction of them) and because there were fewer stories (and therefore, less competition, and thus likely to be more agreement over which of those stories were best).

There are 18 votes (likely to go up even further) and 93 stories this time; the ability of any three people to predict the overall outcome of the vote is going to be vastly worse, simply because there are more good stories to choose from (thus, likely resulting in less agreement) and because there are many more voters (thus making each voter's choices count for less).

Behind Closed Doors had more entries than the minific contests did prior to the influx of new participants, and got nearly twice as many votes as the last pre-influx minific contest got. That's why their predictions have gotten worse - the more voters you have, the more stories you have, the more possibility there is for your top 5 not to match other folks' top 5.

Added to this is the fact that a lot of folks seem to only hand out 1-2 10s, and PP, apparently, only handed out four scores above 7 so far this time. That creates a lot of randomness, because it means that even being in someone's top 5 doesn't necessarily mean you got a very good score, and if you aren't in someone's top 5, you might have gotten a very poor score. If you made my top 5 this time, you got a 10; if you made PP's, you might have gotten as low as a 7. That means that ordinal ranking makes an enormous difference, and moreover, variations in ordinal ranking between participants leads to very large variations in scores; if your 10th favorite story only got a 6, then suddenly making your top 10 is a lot less meaningful, especially if your 40th story got a 4. And even if your 5th favorite story got an 8, that means that most other stories likely got less... which again means that missing someone's top echelons is likely to result in being down in the bulk of stories, score-wise, for that person. By having relatively few really highly-rated stories, they're making it more likely that the high results will seem random to people because stories outside of their top 5 are much less likely to have much weight attached to them; if your top 5 is all 8-10, and your other stories mostly fall in 3-7 with a sort of bell-shaped curve, a story that got, say, 10th from you might have gotten a 6 in a 30 fic competition, whereas a story which got 20th might have still earned a 4. The fact that you fell out of someone's top half, thus, makes less of a difference, but falling out of their top sixth means that you lost two points at least, if not more.

If you are eating 5s and 6s from people who don't put your story in their top 5, and only 8s and 9s from folks who do put your score in their top 5s, then you're not going to end up with an awesomely high final score. Which is what I suspect happened to your story; if it wasn't in someone's top 5, it ended up a fairly middling story for them, which sunk its rating considerably while boosting that of other stories.

Ugh, last post on this topic, then I'm done for today.

Anyway, what I'd really say is this:

Don't let all our talk about votes and weight and all that dissuade you from ranking stories honestly.

If you feel like most of the stories are about as good as each other, having the scores clustered tightly together represents your opinion on them.

If you feel like some stories are much better than others, having the scores more broadly distributed also represents your opinion on them.

If you feel like some are really good, some are okay, and some are weak, then again, using a broad range will represent your opinion on them.

If you feel like the stories fall on a bell curve, with some good stories, some bad stories, and a lot of average stories, then your score distribution should look like a bell curve.

If you feel like the stories have a flat distribution, where there are about as many good stories, average stories, and bad stories as one another, then your score distribution should look like a flat distribution.

If you feel like most of the stories are weak, and a few are strong, then your scores should reflect that; likewise, if you feel like most of the stories are strong, and a few are weak, your scores should reflect that, too.

In the end, if your score distribution represents your honest opinion on them, who cares? Why change it away from something which you feel represents your opinion into something that does not?

The best possible distribution for your opinions is the one that actually matches how you feel about the stories and how well you, personally, think they deserve to do, using whatever criteria you grade on.

3960182
The predictability of finish from certain bellwether reviewers extends far back to the ponychan days, when we'd normally get in the twenties or thirties of participants and actually got more voters than stories.

3956992
I would say that voters can, should, and will discriminate between missing something because of their own inattention and missing something because the story is poorly written... but your Ponychan example seems to specifically disprove that.

I'm going to copypasta what I wrote in the other thread in response to Abion47, then step back. I don't really care how ponies vote but the following is my overly-opinionated opinionation.

3956099
This is well-put.

I don't understand why the scoring system shouldn't be "rate how good the fic is", under the honor system. Any strategizing past that suggests an attempt to game the voting system in order to skew the results toward what you want them to be. You're trying to set the results, not trying to cast one equal vote in a sea of equal votes.

I mean, somepony could decide to vote 10 to anything they thought should win and 0 to anything they thought should lose, and maybe make a 0 into a 10 instead if people all seem to be undervoting something heavily based on their opinion. That would probably be the maximum way to fuck with the numbers. But then that pony's actions would suggest that pony believed that their opinion was more important, or deserved more weight, than the opinion of any other rater. That's hogwash.

and as one last bitch perhaps at least most of us could agree that feedback is more important than winning for buck's sake

Wrong thread (again.) I need to stop having multiple tabs for the same forum open.

3963046

I don't understand why the scoring system shouldn't be "rate how good the fic is", under the honor system.

I think it seems like I'm interpreting things differently from some people, but my overwhelming impression is that this is exactly what everyone has said they're doing.

3964898

3956382

because getting a 0/1/2 would be really really hard since those would be the shit that just barely passes moderation on Fimfic.

Pet peeve: "the shit that barely passes moderation on fimfic" is a myth. About a third my stories fail moderation the first time, and I have to argue, change the description, or find a mod who will approve them. I think I've been accused of violating a dozen of the fimfic rules with one story or another. fimfic moderation isn't about quality, but about following all of the many idiosyncratic rules we've accumulated, as interpreted by whoever takes your story from the queue. Not one of those rules is about whether a story is good or not.

3956441

First, there's shared information being used by all the voters—grammar, spelling, some baseline level of quality stories may or may not have, etc. ... You're never going to escape the shared information, but if you focus on your own unique information, your votes provide more meaning to the overall process.

I have 2 problems with this:

1. If everyone focuses on their 'unique information', we wouldn't be using the shared information. Yet it seems to me the "shared information"--the part of a story that appeals or does not appeal to most people--is perhaps the only thing a story should be judged on. What "unique information" ends up being in practice is mostly "I didn't like your story about vomit because I had the flu while reading it" (which was my reaction to the vomit story), "I hated your story because I hate that ship", "I love your story because POOP", and other stupid stuff.

2. I would be thrilled to believe that "shared information" exists. I think the process of voting is an experiment to see if, working together, we can find if there is any "shared information" (shared judgement) about the goodness of a story. I'm pleasantly surprised to find a correlation between how much I like a story, and how well it places in the voting. I fear focusing on whatever it is that makes your perspective unique--which would usually be "whatever is most eccentric or biased about your perspective"--would stop that from happening.

3964898 "how good the fic is" is not precise enough to be helpful. For instance:

- "Competitive Spirit" made me physically ill because it was well-written. I regretted reading it. Had it been badly written, I would have enjoyed it more (say, a -2 instead of a -5 on a scale of -10 to 10). So which story is "better", a well-written story that makes me sick, or the same story, written too poorly to do so?

- What if a story is supposed to make you feel bad (like, say, "The Cough")?

- What if it's supposed to make you feel bad, but not for any purpose other than making you feel bad (like "Fetch")?

- Is Atlas Shrugged "a good fic"? It was certainly effective at what it was supposed to do. People's reactions to it (how much they liked it) depends entirely (in my extensive experience listening to people rant for and against it) on their ideology.

- Young kids often like stories about red-and-black alicorns, or about boogers. Does that make them good stories?

- Is Stephanie Meyer's Twilight a good story? Or Paul Coelho's The Alchemist? Most of the people who read them, loved them. Surely that means they're good by definition.

- What's the scope of "good fic"? Good fanfic? Good ponyfic? Good write-off fic? Do you penalize it for having all OCs (many people do!), or for being "not pony"?

- You said people should judge based on their "unique information", which I think will always end up meaning "the stuff that you are personally the most stupid about".

My take on the whole matter is that it doesn't make sense to talk much about how to optimize our voting when we don't agree on what we're voting on. Exhibit A is that you think a score should incorporate all the quirky, individual perspectives people bring to a story, while I think it should try to exclude them.

3960362

If you feel like the stories fall on a bell curve, with some good stories, some bad stories, and a lot of average stories, then your score distribution should look like a bell curve.

I think Bradel would disagree with this, on the grounds that it decreases variance. This is dangerous advice, because its true meaning is subtle and different than what most people will interpret it as meaning.

3957707

I mean, not that I'm blaming you guys for it. It's your group, and ultimately you should have the final say on whatever policies are and aren't implemented.

It's RogerDodger's site, and as once-again demonstrated by 3958894, his approach to all such questions appears to be:

- Have minimal rules.
- Have only simple, easily-interpreted rules.
- Leave interpretation of those rules up to each individual.
- Avoid committing the write-off to ideological stances or value judgements.
- Don't try to prevent participants from taking their own ideological stances. The write-off is what its participants make of it.
- Don't try to clean up data if people can't agree on what "clean" means.
- Don't create rules that would give an advantage to people who decided to cheat (for instance, forbidding use of computer analysis to guess authors, or forbidding people from trying to maximize their voting impact).

I think this is the best approach, and I'd be shocked and dismayed if he abandoned it over these voting issues.

3956604

Voting for or against a particular type of story:
Bad Horse has said he does this with comedies. He's also been very explicit about the fact that he's essentially doing it as a shared-information reaction, the same as bookplayer's comments about down-voting "boring genius" stories that draw a lot of praise.

After reading some of the posts here, here is my new-and-improved position:

- If the purpose of the voting is for the best stories to win, I should continue to knock a point off comedies, or any other story that doesn't rely at any point on subtlety; or else I should add bonus points to stories that I think are pretty clear but that I see several other people completely miss the point of. Because I think "best story" means something like "best story among people who understood it, provided it wasn't hard to understand." I would rather see a medal go to a story that was a 9 to the 50% of readers who understood it but a 6 to the 50% who didn't read it carefully, than to a story that was an 8 to everybody.

You want to call me an elitist? Fine, but then you may never complain about the Featured box again, because those are the "best stories" on fimfiction according to your democratic view.

To use my own story as an example, 1/3 to 1/2 of readers didn't understand that "The Sea Pony" was a metaphor for the narrator's own life. I think it's safe to say it suffered a one-point penalty from being misunderstood. (Restoring that hypothetical point wouldn't necessarily have put it in 2nd place, since some of the stories above it were also sometimes misunderstood.)

Yet I don't think that story should be more-easily understood--it's almost as clear as I can make it without bashing the reader over the head with a hammer. I think I hit a good balance between understandable and subtle. Getting half of your readers to understand anything not explicitly stated is doing very well! Subtlety carries an inherent penalty in the voting.

"The Door" wasn't a great story, but it deserved more than a 2.55. It got slammed because most readers somehow didn't figure out that it ended in a rocket to the moon.

I just now looked back over the previous write-offs I've been in. Of the stories that placed in the top three in any of those write-offs, the only ones that I remember people missing the point of were "Elpis" and "Method Acting".

- If the purpose of the ratings is to give authors feedback on how much the average reader liked their story, I should not knock a point off any particular genre. In that case, the deduction for subtlety is part of the information the author would like.

But without any adjustment by people who get the point of a story, anybody who would like a write-off medal will avoid writing subtle stories. I think, based on the 93 stories in this write-off, that this has already happened. Which means that:

- If the purpose of the write-off is to help be better writers, I should add points for subtlety when I see people misunderstand a story. Because the ratings as-is have already taught writers to err on the side of avoiding subtlety.

3966782
Just to point out something from my scoring methodology: I'm not trying to make a bell curve. I don't normalize my scores in any way. They happen to fall where they fall because of my own likes and dislikes compounded with the variety of authors in the pool.

I encourage everyone to vote however they feel like doing so, with a possible emphasis on keeping to your own set of criteria in the hopes of remaining somewhat consistent each round.

3966782
What is dangerous advice - mine, or Bradel's? Or both?

If someone actually feels like a lot of stories are of roughly equivalent quality, then it is best for their scores to reflect that fact, because that way, because then their scores would reflect their actual feelings about the stories. If you feel like a third of the stories are about as good as each other, and all are mediocre, you should be giving them all the same mediocre score; if you feel like stories of considerably different quality are getting the same score, you should try to change how you are grading things so that they don't end up with the same score unless you feel that the difference is irrelevant for scoring (i.e. they're pegging the ends of the scale, and would reduce your ability to separate out other stories).

Increasing variance is only good if you actually think that there is a difference between the stories; otherwise, it is just random noise, which is something you want to reduce.

3966741
Bradel just said that's what we're doing, as Trick Question suggested. He didn't say it was helpful or useful.

More to the point, that's literally all that we can do. Unless we're grading on grammar and spelling, everything else about quality of writing is based on your personal view of what literature is for, what you enjoy in writing, and how you have decided to compare it to other things you've read. We can reach a very rough agreement about some of those things via analysis of ourselves and literature as a whole, but even how much weight we place on that analysis is a matter of personal perspective.

One only needs to look at changing trends in literature over time, or your own changing opinions of literature throughout your life to see how outside tolerance and understanding influences how one thinks of a story, and how that's unique to an individual in a given time and place. It's also amazing the lengths some people will go to to justify personal preferences to themselves as strong suits or flaws in the writing, sometimes without noticing (something I think is much more common in this competition than "I didn't like it because I don't like that ship." It always feels smarter to tell yourself "Fluttershy was acting out of character, and the scene where she confesses her love felt rushed" and not admit you would have noticed neither of those things if she'd been confessing her love to Rarity rather than Twilight.)

So, the answer to all of those questions, whatever we decide in this thread, will always be in practice, "Maybe. How important is that factor to you right now?"

Of course, that doesn't mean we can't talk about it. It means that I don't think the answers we reach will make an overall difference to voting, and they wouldn't make a difference if Roger put them in the rules, either.

3967005
I think you should do the former. But like my own issue, that's going to require reacting to feedback (you'll need to see how often it's being misunderstood) which seems to make one's opinions unpopular. But subtlety is something that you think is good in writing, so you should rate it higher.

3967032

Increasing variance is only good if you actually think that there is a difference between the stories; otherwise, it is just random noise, which is something you want to reduce.

Fair enough. But if you think your discrimination between 5 and 6 is real, but just smaller than your distinction between 1 and 3, spreading your scores out to have a flat distribution (which they will not "normally" have) will probably help make the ultimate voting results more like you'd want them to appear.

3967093

Bradel just said that's what we're doing, as Trick Question suggested. He didn't say it was helpful or useful.

It can't be what we're doing, because it isn't a meaningful statement. "Good story" is too nebulous.

3967181
That's what I meant by the rest of what I said. I agree "good story" is nebulous, and I think people are picking it based on nebulous ideas that often they don't really understand (and which are influenced by a lot of things which even people who think they have a set, nearly objective definition don't notice affecting them.)

3967178
It depends on what your opinion is, really.

If someone really believes that there isn't much variation in quality between the best and the worst stories, then them having a low variance on their scores accurately reflects their opinion.

If someone else believes that there is considerable variation in quality between the best and worst stories, then them having high variance on their scores accurately reflects their opinion.

Your vote will count for more if you have high variance than low variance, but if you don't really have a strong opinion one way or another on the placement of some stories, then maximizing the impact of your vote and accurately reflecting your opinion on the stories may not be the same thing.

I personally feel that there is a lot of variance between the stories, and thus my votes should be very spread out, but some people don't seem to feel the same way that I do. If they don't, I'm not sure that they're really wrong in doing what they're doing, even if what they're doing isn't creating much differentiation between stories in their votes. I'd encourage people not to softball their opinions, though, and I can see why encouraging people to use the full range might encourage them to be more honest about their feelings about stories, but some people seem a lot more ambivalent about such things than you and I.

3957994

Obviously, everyone should be more like me and then the world would be a happier place. :V I don't know what the rest of this thread is about. All I know is that Closing Time is a conspiracy, a group of people are "all in" on it, and I'm mad because I'm not one of them. :(

I don't see anything in the thread about Closing Time.

3967005

- If the purpose of the voting is for the best stories to win, I should continue to knock a point off comedies, or any other story that doesn't rely at any point on subtlety; or else I should add bonus points to stories that I think are pretty clear but that I see several other people completely miss the point of. Because I think "best story" means something like "best story among people who understood it, provided it wasn't hard to understand." I would rather see a medal go to a story that was a 9 to the 50% of readers who understood it but a 6 to the 50% who didn't read it carefully, than to a story that was an 8 to everybody.

You seem to be saying that our goal should be to vote strategically to manipulate the results in a way that better reflects our personal opinions.

This is not a good thing, but I'm not sure that's what you're actually saying.

If you think that the best story is a comedy, and you don't give it the highest score, then you're diminishing the odds of what you consider to be the best story to win.

If you don't think that a comedy is the best story, then clearly it shouldn't have the best score anyway.

There's nothing wrong with saying that more subtle stories are better than straightforward stories; that's a legitimate opinion to have. Or if you believe that more complicated stories are what we should be writing for the write-offs, and therefore you give them higher scores because they are better according to whatever yardstick you are using to rate stories for the write-off because that is the sort of story you believe is "the best" per the goals of the writeoff in your eyes (indeed, this is why I dinged The Arena, because I felt that it relied too heavily on The Lady or the Tiger).

But if you're not giving the best scores to what you consider to be the best stories, and your goal is to make the best stories (in your eyes) win, then you are by definition diminishing the odds of what you consider the best stories to be to win, and not acting in accordance with your goal.

You want to call me an elitist? Fine, but then you may never complain about the Featured box again, because those are the "best stories" on fimfiction according to your democratic view.

The featured story box doesn't actually care about rating (which is how "good" a story is); it is about what is popular. A better example would be the highest rated stories, which often are essentially inoffensive stories which avoid being downvoted (like the Battle of Fort Book and Sunny Skies All Day Long) rather than the "best" stories (The writing on the Wall, Five Hundred Little Murders, Spring is Dumb, ect.)

TD is not in fact contractually obligated to promote Spring is Dumb at every opportunity, but does so anyway entirely of his own free will. Really.

Not to say that the top rated stories are bad (though some of them are, at least in my eyes) or that it doesn't mean anything (generally speaking they are things that a lot of folks will enjoy to at least some extent, and, via their inoffensive nature, are unlikely to be something that folks actively dislike), but that doesn't necessarily make them the "best stories".

This is not necessarily desirable in a competition where the goal is to find the "best story" according to our highbrow standards, though, and we want to show off the "best" story rather than the least offensive one.

"The Door" wasn't a great story, but it deserved more than a 2.55. It got slammed because most readers somehow didn't figure out that it ended in a rocket to the moon.

Honestly, knowing that it was a rocket to the Moon didn't make any real difference in my scoring of the story, because the story still lacked context for me.

I'm not really sure why I was confused about the rocket, though; it was pretty obvious in retrospect. I'm not sure why I thought the blind bag/elevator thing (well, the latter was because it made a loud sound then starting going upwards while playing music, which is totally something that elevators do, while the former... I dunno what the deal with that was. My brain is weird sometimes.)

I just now looked back over the previous write-offs I've been in. Of the stories that placed in the top three in any of those write-offs, the only ones that I remember people missing the point of were "Elpis" and "Method Acting".

Sure, but on the other hand, a story which is complex but doesn't confuse people (The Case of the Cowled Changelings) is probably better than a story which is complex and does confuse people, all other things being equal.

Not all confusing stories are equal; The Sea Pony was pretty obvious in what it was, but never outright said what happened, while Method Acting felt a bit more muddled but it much more directly informed the reader what happened at the end via its central twist, and thus was understood by more folks, I think. Honestly I felt like The Sea Pony was much clearer in what was going on, but then, I'm familiar with the ideas that were presented in it (and have read your other stories) so was well-primed for looking for allegory. If something shouldn't be confusing, and is confusing people, I can see being annoyed about their scores, but if something is confusing people and is doing so because it is, in fact, confusing, that's less problematic.

Of course, here, Daring Do and the Greatest Adventure confusing me made me give it a higher score than it would have received from me otherwise, because I thought it was about something much more interesting than it was and was muddled in its presentation, rather than being about Daring Do just deciding to randomly marry someone for no reason.

3967958
No clue.

3967005
My instinct here is that you're stepping into a Gödel problem. I'm just going to ask: how — especially with no Word Of Author — do you distinguish between stories that are misunderstood through poor communication, and stories misunderstood due to effective subtlety? Surely not every misunderstood story is the latter.[1]

I think voting results are orthogonal to answering that question, so you're not contaminating any information transmission on understandability by scoring stories with that metric; but all you're doing is kicking the "scoring correctly reflects quality" problem up a meta-level. Instead of the correctness of your judgment being based on whether you understood it, the correctness of your judgment is now based on whether you understand why people misunderstood it.

Put another way Tightly related:

This gets back to the problem of depth, as you mentioned in your other thread. Stories get penalized for depth because everyone can understand depth level 0, but if you add depth, some readers will miss it. If you give extra points to compensate for depth which you can recognize, then one of two things is going to happen:

1) You continue giving low ratings to stories you don't understand. This double-penalizes stories beyond your depth level (once because you won't understand them and will rate them low, and twice because you're inflating scores of shallower stories).

2) You give a "subtlety bonus" to all stories you don't understand, which is a blunt instrument that rewards writers for being incomprehensible. This sounds like lazy postmodernism.

Goal #1 is compatible with "reward stories at Bad Horse's preferred depth level", but not compatible with "reward deep stories". (Neither is Goal #2.) So I guess it boils down to what you're trying to accomplish. If you place an extremely large value on a story being at your preferred depth level, it's even a consistent voting criterion, as bizarre as it may sound to lower-depth voters.

Personally, when it comes to voting, I want to reward stories I enjoy, and depth's only part of that. Sometimes I give them high votes even if they're shallow in depth, if I felt they were otherwise excellent. I acknowledge that this is the same problem in the other direction, but at least I'm not trying to meta-out-think myself in the process.[2]

[1] We're talking about behavior as raters/voters here. In the other thread, I'm taking what could be construed as a deliberately anti-depth position, though that's not my intention; it's because, as writers, assuming that we're just below people's depth levels and it's okay that they don't understand is the fastest way to start substituting obscurity for actual depth. I'm at the depth level where genuine depth still feels like it can trace an unbroken line back to the surface. Knowing Latin should reveal Easter eggs in the story, not be the only way to decipher major plot points.

[2] I'd like to note that there is no reference to Footnote 3 anywhere in the text. I'm just sticking something totally unrelated to any of this in at the end.

[3] I understood perfectly well that The Door was about them getting into a rocket. I gave it a low score because it had numerous other major flaws, and I don't think that the successful subtlety on the rocket point improved the story in any way.

3968011 It's simple: I think "best story" means "best story for those who figured it out, given it wasn't too hard to figure out." Everything else follows from that. If 4 people missed the point and gave it, on average, a 6, and 4 people got the point and want to give it a 7, and people correctly identify themselves as having gotten the point significantly more often than they think someone else missed the point when they didn't, THEN I'm saying the correct score for the story is 7, and the only way the story to get that score is for the 4 people who got the point, and are the only ones who have that information, to each adjust their score up by 1 point.

I know people will read that and blah blah about "gaming the system", but I believe it minimizes the error. An honest attempt to minimize the error is not gaming the system.

3968032 I'm not talking about rewarding stories for subtlety. Subtlety isn't an inherently good thing. I'm talking about how to avoid penalizing stories for using subtlety, which is a valid and sometimes necessary tool to accomplish other things.

Nor am I talking just about depth now. People can't tell whether a story is "above" or "below" them, but they don't have to. You just have to identify cases where someone else missed some crucial element of the story, which you thought was explained clearly enough that it's more the reader's fault than the writer's fault, and you have to do that without imagining that some stories had clever critical points that weren't really there. (Like, say, imagining Verdict was set in a dystopian future ). I think people are good at this, and it's not even strongly correlated with depth or IQ or whatever. The danger is probably in making the judgement call on whether a misunderstanding is the reader's or the writer's "fault". (Plus all the litcrit theory that says we shouldn't even be asking that question.)

I'm just going to ask: how — especially with no Word Of Author — do you distinguish between stories that are misunderstood through poor communication, and stories misunderstood due to effective subtlety?

...which is what you said. Yeah, that's a judgement call, and it is the weak point of my idea.

1) You continue giving low ratings to stories you don't understand. This double-penalizes stories beyond your depth level (once because you won't understand them and will rate them low, and twice because you're inflating scores of shallower stories).

Suppose you estimate from reviews that a fraction x of readers missed the point, and you guess it will get a 2 point loss from each of those nx readers who missed the point. For the story to get the "proper" score, the remaining n(1-x) readers need to provide an additional 2nx points, so they each inflate by 2nx/n(1-x) = 2x/(1-x). For x = .5, this is 2 points. For x=1/3, this is 1 point.

Suppose that every voter correctly identifies cases where someone else missed the point of a story, and doesn't imagine people missing points that "aren't really there" (or, alternately, that reader-imputed critical points are equally as valid as author-intended points). Ignore the fact that votes are capped at 10 point. Given all these ideal conditions, the scores would add up correctly and every story would get the desired score.

Since the people who got the subtle thing are the only ones with the information that there was an error, they are the only people who can correct the score. So this is, I think, the only possible general way (meaning, semi-safe for all voters to implement) to make the correction.

Writing all this out makes me less optimistic about it working out correctly in practice.

2) You give a "subtlety bonus" to all stories you don't understand, which is a blunt instrument that rewards writers for being incomprehensible. This sounds like lazy postmodernism.

I haven't and wouldn't suggest doing that.

3968284
Okay, throw out depth. I was getting sidetracked.

I'm basically on board with what you're saying (and as I said, I agree it's compatible with your goal), but I have to raise a red flag here, because this gets exactly back to what I was saying about Gödel:

We just have to identify cases where someone else missed something that we think is obvious, without imagining that a story had some clever point to it that wasn't really there (like, say, imagining Verdict was set in a dystopian future ).

That's a perfect example, because you can't just handwave away "imagining that a story had some clever point to it that wasn't really there". It was perfectly obvious to me that Verdict was a ha-ha-only-serious sugarcoating of a core tragedy, with a high X-value, which would have made it the perfect place to apply your system.

… I had a few paragraphs typed up about this, but you edited your post to clarify and I think we're now in agreement about the weak point. Any system that seeks to augment an individual's impact in order to "correct" the group consensus will, by necessity, magnify errors of judgment.

3968403

Any system that seeks to augment an individual's impact in order to "correct" the group consensus will, by necessity, magnify errors of judgment.

Any system that requires more judgements will increase errors of judgement (sorry, I still like it with two 'e's, like dodgeball or escapement). But it may reduce other errors more than it magnifies errors of judgement.