Sunday, April 9, 2017

Slating Analysis: 2017

How Many Slate Voters Nominated in 2017?

Now that the 2017 Hugo Awards Finalists lists have come out, we can estimate how many slate voters there were. By our calculations, there were between 88 and 118 of them. This is just slightly higher than Mike Glyer's estimate of  "80 to 90". When the detailed statistics are available in August, we'll make a more precise estimate, using the methods we used in our article Slate Voting Analysis Using EPH Data: 2014-2016.


The chart shows our best estimates for the number of voters from the two different "puppy" slates over the past four years. Note that these are people who voted in the nomination phase, not people who participated in the final vote. Contrast that to the total ballots:


Despite casting a tiny fraction of the ballots, the slates had an outsized influence in previous years, grabbing most of the nominations and sweeping many categories outright in 2015 and 2016.

To properly interpret these charts, bear in mind that the Hugos are awarded in a two-step popular vote. In the nominating phase, anyone can vote if they were a paid member for the previous year, or a paid member for the current year, or if they were preregistered for the following year. The final vote, however, is only open to members who are registered for the current year.

Here are some key things to take away from these charts:
  • 2014 was the last "normal" year in terms of nominations and the final vote. The small slate presence made little difference that year, although it was enough to encourage the organizers to try harder the next year.
  • In 2015, when the slates swept the nominations, thousands of fans turned out to support the awards, pushing the final vote totals into the stratosphere and burying the slate nominees under No Award.
  • In 2016, people who'd paid for 2015 were still eligible to nominate. That boosted the nomination totals for organic fans and slate voters alike.
  • Despite the huge nomination turnout in 2016, the slates swept the ballots again. Discouraged fans didn't buy new memberships, so the final vote was actually down from 2014.
  • 2016 was the last year under the old nomination rules. The new rules make it harder for slate voting to sweep entire categories.
  • In 2017, the Sad Puppies have disappeared, and the Rabid Puppies are a shadow of what they were. The Rabid organizer, citing the new nomination rules, urged his followers not to register and then produced a mini-slate with just one nominee in most categories. 
  • The fact that the 2017 nominations are up 15% over 2015 is probably due to the greatly improved software for submitting nominations as much as anything.

Did the New Rules Make a Difference?

The biggest difference from the new rules seems to be that they made the slating groups give up. The big difference in 2017 is due almost entirely to the fact that the slate organizers didn't even try, although it does appear that EPH (the E Pluribus Hugo voting system) eliminated three slate nominees: one each in Best Relatd Work, Best Fancast, and Best Fan Writer category.



APPENDIX: Details of Analysis

For those who want to know exactly how we came up with these numbers, read on.

Slate Estimate for 2017

We divided the categories into four different cases:

Case 1: Slate Nominees 1 → Finalists 0

This sets an upper bound, based on the 6th-place vote total. That is, if the slate nominee had gotten more votes than that, then it would have been on the list.

Case 2: Slate Nominees 1 → Finalists 1

This sets an upper and a lower bound. This is simply because if a nomination made it into the list it couldn't have received fewer votes than the minimum, nor could it had received more votes than the maximum. 

Case 3: Slate Nominees 2 → Finalists 1

Here EPH comes into play. This sets both a lower and an upper bound. The total votes could be no less than the number of votes for the #6 finalist (or else there would have been 0, not 1) but the total points could have been no more than the points for the #5 finalist. 

We won't know the actual number of points for the organic nominees until the complete statistics are released after the Hugo awards have been presented, but (as we'll discuss below) we can estimate them. As for the slate nominees, number of points is exactly half the number of votes. From that, the upper bound is simply double the number of points the #5 item received.

Case 3: Slate Nominees 2 → Finalists 2

This also sets an upper and lower bound. The upper limit cannot be more than the most nominations any finalist received. The lower bound is twice the points of the #5 item.

Hostages

The logic above only works for nominees that would not have gotten any votes from anyone except the slate. "Hostages," nominees that would have gotten some votes in their own right, can only set the upper bound, not the lower.

We ignore the effects of disqualifications and withdrawals. All that matters is whether the slate nominees had enough votes to make the finalist list--not whether they actually ended up in it.

Results

CategoryNom MinNom Max#5 PtsSlate NomSlate FinalHostageSlate MinSlate Max
Best Novel15648013110FALSE
156
Best Novella16751114211TRUE
511
Best Novelette742686911FALSE74268
Best Short Story871827711FALSE87182
Best Related Work884248721TRUE
173
Best Graphic Story712216100FALSE
Best Dramatic Presentation, Short Form911937111TRUE
193
Best Dramatic Presentation, Long Form240103020211TRUE
1030
Best Editor, Short Form1492299910FALSE
149
Best Editor, Long Form832016811FALSE83201
Best Professional Artist531434322FALSE87143
Best Semiprozine804345411FALSE80434
Best Fanzine531593811FALSE53159
Best Fancast761095921FALSE76118
Best Fanwriter801526121FALSE80122
Best Fan Artist391213522FALSE69121
Best Series12932510610FALSE
129
John W. Campbell Award for Best New Writer882556611FALSE88255

"Category" is the Hugo category.
"Nom Min" is the number of votes the #6 nominee received, as reported by the Hugo committe. "Nom Max" is the number of votes the #1 nominee received.
"#5 Pts." is the estimated number of EPH points of the #5 nominee, computed as described below.
"Slate Nom" is the number of nominees in the category proposed by the slate.
"Slate Fin" is the number of finalists in the category won by the slate.
"Hostage" indicated whether we think the slate nominees would have gotten votes from anyone outside the slate.
"Slate Min" the lower bound for the number of slate voters to have produced this result.
"Slate Max" the upper bound for the number of slate voters to have produced this result.

The largest min is 88 and the smallest max is 118, so we predict the actual number of slate voters fell between those numbers.

Conclusion

There were no fewer than 88 slate voters in the nomination phase this year and no more than 118 of them. The lower bound is firm, but the upper bound depends on our estimates of votes and EPH deflator for the #5 items. 129 is an absolute upper bound.

Interpolating the 2017 Nominations

Published Data

The Hugo Committee published per-category results giving just the number of votes for the #1 and the #6 finalists. For our analysis, we need both the number of votes and the number of EPH points for each finalist.

Estimated Votes

To estimate the number of votes, we relied on the fact that organic nominations generally follow a power-law distribution. We computed an exponent for each category and used that to interpolate the number of votes for positions 2, 3, 4, and 5.

That means that for some constant, m, we can estimate the votes at rank i as


Since we know the number at rank 1 and rank 6, we can compute m as follows:


Looking at historical data, this seems to work fairly well most of the time (usually giving numbers within 20%) but it can be way off.

Estimated EPH Deflator

To get the EPH points, we simply took the median EPH deflator for each category, using the numbers from the 2014, 2015, and 2016 Hugo Nominations Organic Estimates, and applied that to all the ranks for that category. For the two Dramatic Presentation categories, for which we have no EPH numbers, we used the deflator for Best Novel.

Missing Category

For Best Series, which did not exist before 2017, we just used the EPH deflator from Best Novel.

4 comments (may contain spoilers):

  1. The unavoidable conclusion is that some slate voters left Vox Day off the ballot as Best Editor. And doesn't that seem surprising?

    ReplyDelete
  2. How do you figure? I get the same number in all categories. Some of the 2016 data suggested not all of his guys nominated him, though.

    ReplyDelete
  3. Enlightening as always!

    I assume Mike means the difference between the minimum of 88 and VD's score of 83 (assuming he was in sixth place).

    I'd assume, though, that there would be a bit of wobble even around the non-hostage vote. e.g J.Mulrooney might have 1-5 pals who aren't Rabids but who are willing to spend some money on voting for their friend.

    ----

    For the slate_nom=1/slate_final=0, do the seventh place estimate figures using the power-law fit with Rabids being in 7th place? (i.e. around 80)
    [I guess I could just check myself! :) ]

    ReplyDelete
  4. Ah, I see. You're assuming that VD got fewer nominations than any other long-form editor. These numbers suggest that he was at worst in 5th place, not 6th.

    You're definitely correct that the assumption that anyone who wasn't a hostage got zero non-slate votes is questionable. That would have the effect of causing us to overestimate the number of slate voters.

    For your last question, there are only three 1/0 cases, and the numbers for position 7 are 141, 143, and 119, respectively, so that suggests that the rabid nominees in those categories are further down the list than position 7.

    And thanks for the kind words!

    ReplyDelete