Said RogerDodger on the "Closing Time" thread:

This is actually a big concern I have with the prelim rounds as is.

Actually filling in the form takes all but a few seconds, and random votes are very much worse than no vote at all. If one is inclined not to read any of the entries, then the current system of forcing that person to fill in a dodgy vote just makes things worse. It would probably be best to move filling in the prelim records from mandatory to highly recommended.

This is just for brainstorming ways to incentivize not only entering a preliminary round vote, but actually considering what you vote for. The final decision is, of course, totally up to our benevolent overlord RogerDodger.

Maybe offer a small number of bonus points for something like entering a preliminary vote where one of your top three ranked stories makes the final round? (Forcing people to think carefully about which ones on their ballot are best.) This could be done either with mandatory or highly recommended participation.

The downsides to that would be that it could become a cycle (a story is getting good reviews, so people rate it higher hoping that means it will make the finals, regardless of how they would normally vote) and that some bunches might just not have anything that's going to make the final round. But if the point value was negligible, hopefully people wouldn't see it as too unfair or be too concerned with gaming the system.

On the upside, it might encourage people who didn't write to participate in the preliminary round to offset some of their point decay.

Anyone else have thoughts or ideas?

4042248
I actually have an interesting one:
In the prelim block, there's the setting to organize in order from best to worst. Perhaps if there were a way to include small text inputs below each story spot within which you should give a short review of the story - proof of read - it could be better weighted to actually involve reading.

The authors/stories that would fail would be the ones with summaries/reviews least relevant to the fic, ergo if they skim, they get a penalty. It encourages reading and participation, and writing three sentences about a fic isn't too much to ask, is it?

Of course, full-on reviews in the thread would be preferable, but this would cut down on the 'random votes' and the skimmers. Therefore, if you're willing to make the small effort to review, you're going to be rewarded with a better chance at moving on.

This is all just speculation, of course. I've no idea how the mini-reviews would be implemented, but it would be an effective way to show who's doing their work and who's just taking potshots.

My only concern is this possibility. Somepony submits a story without realizing they will be required to read up's of 30,000 words that week, or their entry will be rejected. I was able to devote time to reading this week, but I didn't realize when I entered that I would automatically lose (and my story would not even be read and reviewed by most authors) if I didn't have the time to review others' works.

Without the preliminary round, voting has always been optional. Moreover, there is no way to "n/a" one of the stories if you can't finish it or don't feel qualified (or don't know how) to rank it relative to the others.

Those, however, are small problems which do not by themselves invalidate a good solution.

4042283

This is all just speculation, of course. I've no idea how the mini-reviews would be implemented, but it would be an effective way to show who's doing their work and who's just taking potshots.

The downside I see with this one is that someone would have to take a look at the mini reviews and maybe even the stories to make sure they line up. I hesitate to suggest anything that can't be automated...

4042330
Hrm. I can't think of a way to work that through then.

Good idea, wrong implementation. :\

Admittedly I skimmed the "Closing Time" thread, but... this seems really baffling to me. We've never had concrete concerns (outside the theoretical) that anyone was just slapping down random numbers on their voter form for the final round. So I was a bit surprised to see Roger state his concern about this happening for the prelim round. I guess, Devil's Advocate, that when the prelim round is made mandatory, this creates a motive to submit a BS ballot, whereas previously no motive existed. But IDK, color me naive, but with how "srs bsns" a lot of us have been discussing voting systems and scoreboard points and all that other stuff, it seem incongruous to also say "and we're the type of people to slap random numbers on a ballot."

4042248

Maybe offer a small number of bonus points for something like entering a preliminary vote where one of your top three ranked stories makes the final round?

Biases voters toward herd mentality. Voting for controversial favorites can penalize you.

Dunno, I might be taking the unpopular position here. I know that voting has always been optional, and I know that not everyone always has time to read all the competitors (this is actually one of the main reasons I've skipped the last few rounds). But in my mind, the benefits of a mandatory prelim round outweigh the cons. It helps cut down the field of finalists, which helps everyone. It repays the kindness of your peers; you read my fic, so I'll read yours. It encourages the prelim voter to leave reviews (though I'd still assume these should be optional), thus helping to ensue that everyone gets a fair amount of constructive feedback, even if you were eliminated early on. The prelim round should, in theory, be small enough that it consists of a reasonable amount of reading for all participants (if it's not, then it should be scaled back).

Presuming that not everyone feels as strongly as me about mandatory prelims, then I'd walk it back just one step and ask, "Given the self-evident benefits of prelim voting, if we choose not to make it mandatory then is there any need to further incentivize participation in the prelim round?"

4042248

As an outsider who eventually would like to participate in one of these competitions, I feel pretty intimidated by all this talk of a seemingly quite complex voting system.

I just wanna know if my entry was good, y'know?

Sorry if this seems like a useless post but if you want to keep this whole thing going, it needs to be presented as a quite simple system. Or maybe I'm just being silly. Either's possible.

Presuming that not everyone feels as strongly as me about mandatory prelims, then I'd walk it back just one step and ask, "Given the self-evident benefits of prelim voting, if we choose not to make it mandatory then is there any need to further incentivize participation in the prelim round?"

I was simply going by the assumption that it was originally mandatory for a reason (validity of results? concern about low turnout?), and that some method of increasing the voluntary turnout while still encouraging careful selection might mitigate whatever the original concern was.

For all I know, it's no big deal. That was just my thinking.

I've never much liked the idea of people entering the writeoff and not participating in the voting. If you're asking all of us to read your stories, the least you can do is offer the same courtesy.

Frankly, an easy way to deal with people not caring is just throwing out their votes if they don't click on the stories/don't change the ordering of the stories in their ratings (the odds of the stories being correctly sorted to begin with are 1 in 5040, so the odd instance where they are correctly sorted by chance is sufficiently rare this wouldn't be a big deal, and/or Roger could PM them and ask).

I think this is mostly worrying over nothing. Don't advertise the idea that people could do this, and most of them won't even think to do so.

Providing any kind of incentive for participating runs into the same problem. Filling in the form doesn't take long, so people can simply make dodgy votes to claim whatever incentive is there.

The current system is useful in the case of conscionable participants. I know a lot of you only get to work when there's some kind of deadline, so without an incentive in place some of you might skip the prelim rounds for one excuse or another yet wouldn't dare compromise your integrity by submitting a dodgy vote. However, in the case of unconscionable participants, it's bad because it encourages them to submit dodgy votes, which are worse than no vote at all.

4042248

Maybe offer a small number of bonus points for something like entering a preliminary vote where one of your top three ranked stories makes the final round?

This is not an incentive for participating but for making your votes as close to the mean as possible. This turns the voting into a Keynesian beauty contest, which is not desirable.

4042420
I'm concerned about dodgy votes because they have a very large impact on the outcome. Remember that each story only gets 7 votes (in this case), which is a very small sample size. If just a couple of those are dodgy, it could skew results dramatically.

4042705
Someone inclined not to read the entries could still shuffle the form around a bit before submitting it.

4042777
I wasn't suggesting throwing them out of the competition, just throwing out their prelim round vote. And not advertising the fact that the system functions in this way.

Of course, you could throw them out of the competition as well, but that isn't necessarily, well, necessary (and if you don't separate out those who did not participate in the prelim round from those who did, they would be none the wiser regardless).

How about measuring the scope of the problem before we decide how hard to come down on it?

Is there a way to put a checkbox at the bottom of the form, something like "I decline to rate these entries"? (For maximum guilt-trip, you can add afterward in small text, "The writeoff is an all-volunteer effort. You are allowed to decline judging, but we encourage you to give something back to the community that's rating and reviewing your stories.")

Then you can make the form mandatory, but the sort of entrant who would be incentivized to submit garbage data is now incentivized to just click the ticky box instead. If null votes are better than garbage, that seems to me to align the incentives properly.

If large numbers of people actually do click the ticky-box, then we should definitely revisit the conversation about mandatory effort, but my prediction is that the rate of problem voters is low.

4042883

but my prediction is that the rate of problem voters is low

Or nonexistent. If random voting happens, then yes, it's going to be a major problem. Though, then again, with a bunch of the reviewers apparently using the new system to get multiple story ballots to rank-order (myself included), a few people turning in junk data is probably going to decrease in importance—but it may still make a few edge-case stories lucky or unlucky.

My guess, though, is that with the general amount of involvement we see, and the fairly low cost of completing one ballot, and the fact that essentially everyone participating is interested in making sure the write-off does yield valid results—my guess is that with all that, nobody here is very likely to be doing junk voting anyway. Then again, I have what might be considered a naive level of belief in the fundamental goodness of humankind.

4042883
Honestly, this is probably the best solution.

4042777
I did forget about the sample size being smaller. Mmph.

I dunno, this just seems... Okay, hypothetically we make a new rule. "NO DODGY VOTES! SHITVOTING = PERMABAN!" Roger has full and unrestricted access to all prelim round and final round voting ballot data, so pretend that he looks around for any dodgy votes so that he can manually ban those users. How exactly does he determine what vote is "dodgy?" Something that deviates outside the norm? Something that just votes 1-2-3-4-5 down the ballot? Is there a way to detect a dodgy ballot?

The prelim round will be a smaller sample size, but the final round still has a relatively small pool of 20-30 votes, enough that each vote has substantial sway. If we can't systematically prevent gaming the final round, then it's sort of a moot point to discuss how to prevent gaming of the prelim round, innit? I think our process of voting in a winner can only be as accurate as our voters are honest. Garbage in, garbage out.

4042883
I think this is the best approach.

4042883
I like you. You're a clear cut solution maker. Thank you for your input.

I approve of this man's methods.

4042420

whereas previously no motive existed

Except prelim rounds have always been contingent for participation, unless I miss my guess. c.c

Also wow, horizon's totally on the ball.

4042883

Well, if you have the option to decline rating certain entries, then while we're not too concerned about dodgy voting and stuff due to integrity and all, wouldn't it be easier to make it optional to vote in the first place? If someone who ticks most of the check marks because the had no time does so, then as you guys said, it's a no vote situation; but everyone is alotted a certain number of stories to vote on. What if lengthier stories get the drop because they're more time consuming to read? They don't get any vote at all, and though it is argued that that is better than garbage voting, that still means that those stories are put at a disadvantage, since the tick mark implicitly counts against them.

Perhaps a second wave of prelims where pre-determined volunteers read up and vote on the stories that didn't quite get the same kind of attention as the others. I'm new to this whole system, so I'm not entirely sure if equal exposure was the point of the story voting division of labor, but if it was, this might help taper the odds that alot of tick marks appear on lengthier or more complex stories, especially when the word count is higher, season is more tight, etc.

4042883
in brief: responded-to is reasonable.

in more detail:
I will agree that this proposal, an opt-out, makes the most sense as a means to provide disincentive to submit dishonest ratings. It relies, however, on the expectation that the preponderance of participants will continue to behave 'properly,' submitting a completed form; remember that the stated goal of prelims is to cull the field of submissions to a more manageable selection. Given the swelling of participants, can this assumption be expected to hold? Granted, requiring the overview of (in this case) seven submissions as opposed to any number up to thirty-six is a significant reduction, but may still be perceived as an overbearing task when there is now an option to 'opt out' and not be removed a priori. I have no clear ideas on how to balance these aspects, but I raise it as it has not yet been mentioned.

Similarly, it may be worthwhile to examine the motive for so many participants. What were the features of this group that made it effective, say, four events prior, and what may have led to the situation of seventy-plus minifics and thirty-plus short stories for the respective past two events. I posit that the driving motivation is exposure and feedback, with a benefit of a prompt to spark an idea. The incentive to participate, then, is in terms of being read. To what degree the feedback is used to grow as a writer is likely beyond the scope of this discussion. If this is accepted, how can this be considered to play into the various rounds of voting? Again, I have given this little thought and have no good suggestions.

As a final note, as read above, there is the overarching desire 'to know if my fic any good.' In the past event, there was a somewhat-heated discussion as to the various metrics one might employ to answer this question, and it stands to reason that, so long as each participant is consistent in their metric per voting round, results will be fair, but is not a guarantee of any one aspect (or suite of aspects) of quality. Can writeoff events find a 'good fic' balance for sets of an arbitrary size? Should it try to?

Quick specific responses:
4043615

What if lengthier stories get the drop because they're more time consuming to read?

My suggestion is that there's a single opt-out box which N/A's the entire slate of seven. Either you commit to reading and ranking your preliminaries, or else you welsh out on the whole deal and the rest of us at you. No offense to our friends from Wales.

Allowing the N/Aing of stories one-by-one has its good points and its bad points (based on the stated goals of the preliminary round, I think I'm on the side of overall bad), but it's a totally different thing from this discussion, which is what to do about hypothetical authors who want to submit stories but refuse to read them.

4043658

It relies, however, on the expectation that the preponderance of participants will continue to behave 'properly'

Hence my suggestion that we see how it works in practice. I believe that the expectation you point out will hold; but we don't have to commit to a decision until we test the hypothesis. Doing it that way for a round lets us gather data and skip all the arguing about human nature, and then if it works in practice, then we can point to that as evidence that it'll keep working. If not, we get to return to the discussion of how to save us from ourselves, knowing that it's necessary.

Horizon's idea is perfect for handling people who just don't want to participate for whatever reason.

If it's not already doable, another thing to consider is allowing people to do multiple 'pods', at least for cutoff purposes. I think I'll probably end up reviewing every entry in this writeoff because I both have 2 weeks to do it and thus the time to read every entry for enjoyment purposes first, and then go through a second time with intent to purely review, and I find that -wonderful-.

But I digress : Multiple 'pods' would allow for more votes per story as the finals go, with the potential downside being some users have greater impact than others, but given the prelims are a simple ranked preference even TD can't nuke the vote being his usual disgruntled ZEROES FOR EVERYBODY self

TD, thank you for always being a convenient target for ZEROES FOR EVERYBODY

4043892
You can actually do this now! If you go back to preliminary voting after submitting your first set of 7, you can request another grouping of stories to rank.

4042883 4042248 4042288 4042705 4042777 4042972

As far as the participation in prelims goes, making the simple voting on a list of a few random stories mandatory is okay with me. People shouldn't expect to enter a contest like this and sit back and watch everyone else do the work to score and review everything.

As far as trolling the vote goes, we all know this group is full of trolls. Even so, there isn't much trolling they can do in the open voting rounds with a single ballot. In the prelims, unless people consistently prelim vote on more than the one mandatory ballot, trolls could significantly alter the results. That's why I would like to have the 0-10 scoring sheet be present in the prelim scoring the same as it is in the finals. Only for the small group of stories per ballot, but still have the ability to score everything equally across the entire final range of scores -- this makes a huge difference for someone like me who plans on voting my way through the entire list during the prelims.

My only problem with the current prelim voting is this:

I got a pretty bad selection of first and second ballot picks. Out of those 14 stories, only two or three were stories I felt should make it to the finals. I would like to see a way to score or indicate the individual stories in the list that I feel should make the final while also ordering them in preference or total score like the final round. If I get a ballot full of awesome stories, I'll indicate everything as final worthy or score them all high -- perhaps I got lucky and got the cream of the crop all on one ballot. If I get crap, I don't want to have my vote on the prelims count the top story that was the best of the bad ones the same as the top story that was actually good on another ballot set.

4043807

My suggestion is that there's a single opt-out box which N/A's the entire slate of seven.

Then the entire list would get scrapped, stories both long and short on the list. And then it is as if said person did not bother participating in the voting. With the amount of reviews I see people doing and saying that they're intending on doing, I guess I shouldn't worry too much about stories not getting review exposure. After all, those who want to review will review whether it is mandatory to vote or not.

I was still a little worried about groups of stories having a disadvantage because people who got their shares just don't participate in the voting component for various reasons. However, if there are people who want to pick up some extra brackets, those who voted not to participate could pass their brackets on to those willing and with the time to pick up the brackets so that they don't just get dropped off altogether.

A lot of thoughtful discussion here. It's nice to see people taking such an active interest in the group

I agree that 4042883 's plan is probably the simplest and best solution, though I also think 4044215 has a good point about the necessity for an ability to say "I think all of these should make it to the final round" or "none of these should make it", or something in between.

4043807
Another thing worth noting is that, as of right now, we have 43 completed ballots with more than half of the prelim time left. That's quite a lot of votes.

Given that a number of folks appear to be intending to review everything anyway even prior to the final week, I'm not sure how much of a concern this really is, though it might end up weighting things more heavily towards the folks who DO, in fact, rate absolutely everything.

Which is fine, honestly; you would expect the people who actually care enough to spend a bunch of time doing it to be the ones who are, at the very least, interested enough to read all the stories.

If it ends up a cluster when we try it, then yes, we can reconsider.

4043892

ZEROES FOR EVERYBODY

You're welcome!

Though in all fairness, that's more of a minific round thing; few people submit storyless stories in the short story rounds.

4044215
The issue is not whether or not people will troll; the issue is whether or not lazy people will simply vote at random/vote in the order the stories are given to them so that their story is qualified for the next round. Random votes are better than no votes.

There's nothing we can do about active malfeasance short of banning someone; only one person has ever tried it.

As far as scoring goes: the point of the preliminary round is to make sure that a smaller number of stories go forward. With only a very small number of stories graded by each person, differences in grading become much more significant. That's why there is the forced ordinal voting here; if someone grades all stories on a 6-10 scale for the final competition, that doesn't negatively impact the results. If someone does that in the preliminary rounds, then they are likely to distort the votes, and someone who got rated by me, for instance, would be at a disadvantage compared to someone who got rated by someone who gives out nothing lower than a 6 to anything which is not really dire.

Ideally, if we have a large enough sample size, the fact that some stories lost out to some other stories isn't going to be that big of a deal. I'm also not quite sure how the back-end calculations work; if something comes in 7th in a field of stories where the first six all passed through, that last place placement may not be as bad for it as if it came in 7th in a field of stories which all failed.

Not sure how it works on the back end though.

4044255
The purpose of Horizon's suggestion is to prevent people from voting at random just to make sure that their story is potentially qualified to go on to the next round.

The point of having as large of a pool of voters as possible is to give us as much confidence in the results as possible; probably somewhere around a dozen of us will likely read and review every story, but a larger sample size would be preferable.

4045094

So a "no vote" would still have his/her story advanced? Might as well not make it mandatory to vote then. Just speaking for myself here, I would have much less an imperative to vote were I not:

A) Given a relatively short list to read

B) Booted from any chance at finals without voting (even if my story isn't exactly winning material, it'd be shooting myself in the foot not to vote and get auto-cast out as a result).

That's just me though. I don't mind giving feedback in the form of votes, but I imagine I would procrastinate a bit more if the clock was not ticking towards disqualification. But if most people are more punctual in their mindset, then this is more a personal problem then an issue at large.

I just had a thought.

I can only speak to the reviewers, who are most transparent and vocal about their reading, but it seems like the prelim round only serves to extend the voting an extra week. Everyone who's reviewing is using the prelims to get through everything as it is (I've always done that myself, and reviewed anything that didn't make it). Unless there's a cadre of quiet casual participants who would prefer not to have to read all the stories if they can manage it, it seems to me like just extending the voting period a week when there are too many words would suffice to solve all the problems.

You get more time to read (which was the main issue with All In), the voting and reviewing progress apace, and about the only thing different is no built-in randomization (because I know some people were concerned about all the reviews going in order). Am I totally off base here?

4054623
Looking at the review spreadsheet data:

3 reviewers reviewed all 36 entries
4 reviewers reviewed around half of the entries
14 reviewers reviewed between 3 and 10 entries

So you're a bit off base. Most people are just doing their assigned sets.

4054663 I'm concerned that if I request a second set of 7 stories, I'll be booted from the next round if I don't rate all of them, too.

...though I'm probably going to be booted anyway.

4055731
The check is if your number of filled records >= your number of entries, not that all your records are filled. It is a bit confusing, though.

4056332 What counts as a "filled record"? Does that mean someone who submits 2 stories has to do 2 rounds of reviews?

4056341
I think that was what was previously discussed, yes.

Which raises a secondary question about protocol if an author completes some records, but not as many as the number of stories submitted. Do their top rated stories continue into the next round, up to the number of records they completed (which seems like the fairest option to me), or do all their stories get rejected, or is there some other result?