Thursday, September 22, 2016

Slate Voting Analysis Using EPH Data: 2014-2016

Using the EPH (E Pluribus Hugo) data provided to the WSFS Business meeting in Kansas City, it's possible to make much more accurate estimates of the number of slate voters in 2014, 2015, and 2016.

We find that their numbers roughly doubled each year, starting from about 100 in 2014 and climbing to about 400 in 2016.

We also find that their discipline across categories improved considerably.

Based on the drop in their voting rate on the final ballot, we cautiously estimate their numbers may fall back to 200 during the 2017 Hugo nominations.

We find that the impact of slates has been almost entirely limited to keeping worthy works off the final ballot.
  • Only one Hugo winner (Best Professional Artist, 2016) owed its place on the ballot to slate nominations. No other candidate who chose to remain on the ballot despite an unwanted slate nomination actually needed the slate nominations to get there.
  • Only two candidates (Black Gate in 2016 and "Goodnight Stars" in 2015) who declined the nomination in response to being slated would have made it to the final ballot without the slates.
This means the slates are not, for the most part, succeeding in getting awards for their own favorites, nor are they successful at frightening legitimate nominees into withdrawing, nor are they successful at inciting ordinary fans to vote against worthy nominees. The only thing they've succeeded at is reducing the number of quality candidates that fans can choose from on the final ballot.

Update (10/2/2016) We also find that as many as 85% of the Rabid Puppy slaters defied their organizers and refused to nominate some of the "hostages." See When Slates Collide for details.

Background

For the past four years, the Hugo Awards have been targeted by organized groups trying to influence the awards to make a political statement. Wikipedia gives a decent overview of the issues and history, but the essence of it is that these groups have been able to hijack the Hugo Nomination process by taking advantage of a defect in the nomination process. They are unable to hijack the actual awards, due to their lack of numbers, but by filling the list of finalists with low-quality works, they've managed to prevent awards from being given at all in a few categories.

The membership of the World Science Fiction Society, which administers the awards, has been working to to change the nomination rules, and to make intelligent decisions on some of the proposals, it helps a lot to know how many slate voters there actually are.

What is a Slate?

I'll define a slate as a list of up to five items per category which a) are proposed for political, not artistic purposes and b) are meant to be voted on as a block, regardless of whether people know anything else about them. Normally, Hugo voters nominate things based on their merit: this novel was good, that artist drew some great covers, I really liked that TV episode, etc. Slate voters nominate things based on whether they agree with the politics of the slate creator.  

What is a Slate Voter?

For our purposes, this is anyone who voted for all the items on a slate and nothing else. Many more people might be influenced by a slate, but we're only trying to measure the number of 100%, hard-core, slate voters. Everyone else is an organic voter. Even someone who voted for 4 out of 5 slate nominees in a category is an organic voter for our purposes here.

This makes the math easier, but it does mean we're underestimating the total impact of the slates. To reduce that effect, we compute the number of slate voters separately in each category. That is, we don't require a slate voter to have voted the slate in every single category. 

As we'll see, for a great number of slated works, once you remove the slate contributions, there are only a handful of votes left. Slate-influenced voters who don't vote the whole thing probably have an influence, but it's small compared to the ones who vote the whole list.

Previous Work

At the end of April, we did an analysis of slate voting for the 2016 Hugo nominations, in which we concluded there were only about 205 "Rabid Puppies" and zero "Sad Puppies" this year. The numbers below will show something closer to 400 Rabid Puppies and perhaps 20 Sad Puppies. The factor-of-two miss resulted from two things: First, there was an error in my math. The algorithm we used should have predicted something closer to 300. Second, one of the underlying assumptions was that organic voters would vote in the same distribution as in past years, but, in fact, the top 15 nominees had more votes across the board than that model would have predicted. This suggests that ordinary fans paid more attention than usual to what other people were saying and tended to concentrate their votes more on works that were more popular.

We did accurately predict who had declined the nominations, so it wasn't a complete miss.

What Did We Do This Time?

The EPH data released to the public at MidAmeriCon II in August contains a lot of information that can be used to estimate the number of slate voters, given the very strict definition above. We extracted that information, calculated the number of Rabid and Sad Puppies for the years 2014, 2015, and 2016, and posted the results on this web site. 

Details of the calculations are in an appendix.

Analysis by Other People

Brandon Kempner, at Chaos Horizon, estimated there were about 440 Rabid Puppies in this year's nomination process. 2016 Hugos: Some Initial Stat Analysis, August 20, 2016.

Aaron Pound, at Dreaming About Other Worlds, produced his own per-category estimates for the numbers of Sad and Rabid Puppies and produced his own version of what the finalists would have looked like in the absence of slates. He estimates roughly the same number of Rabid Puppies as we do, but he gets a higher number for Sad Puppies. We attribute much of the difference to the fact that we're using a very strict definition for "slate." Random Thought - 2016 "What Could Have Been" Hugo Finalists, September 20, 2016.

Aaron has also done a very lengthy, category-by-category, work-by-work analysis of why the results came out the way they did. Not in terms of numbers, but in terms of details about the works and the people who created them. This is a great read for anyone who's really curious as to why some of the slated works seem to have done a lot better than others. Biased Opinion - 2016 Hugo Awards Post Mortem, September 13, 2016.

Results

Total Slaters

If we look at the estimated slate votes per category across the three year period (combining the Sad and Rabid Puppies for the moment), it's very clear that the slates have greatly increased both their numbers and their consistency.

Click to Expand
Best Editor, Short Form is an outlier because the Rabid Puppy slate only had a single nominee in this category. The algorithm is therefore calculating how many people voted for Jerry Pournelle and no one else. This offers a way to estimate how many slate-influenced voters there were, since Jerry wouldn't have been in the top 15 without the Rabid slate calling him to people's attention this year. That suggests that there were 300 hard-core slate voters who voted the straight slate in all categories plus another 150 slate-influenced voters who voted selected things from the slate.

In 2015, they had as many as 250 slaters in the novel and novella categories, about 200 in the middle categories, dropping to 150 in the art and fan categories. In 2016, on the other hand, they were very close to 400 across all the written works and stayed above 300 in all categories. (After we made the change to reflect the split Rabid Puppy slate in that category.)

In mathematical terms, excluding Pro Artist 2016, the normalized standard deviation (S.D. divided by the mean) fell from 42% in 2014 to 20% in 2015, to 15% in 2016.

If we consider only the Rabid Puppies, they improved from an average of 130 per category to 370, and their normalized standard deviation fell from 19% to 14%. In other words, they almost tripled their numbers while sharply improving their discipline.

At the 2016 WSFS Business Meeting at MidAmeriCon II in Kansas City, MO, Dave McCarthy said that he thought 2016 should have been an easier year for EPH than 2015, but these numbers say that 2016 was the bigger challenge.

Rabid vs. Sad Puppies

The next two charts separate the Sad and Rabid Puppies for 2015 and 2016.

Click to Expand
Note that in the 2015 chart, the algorithm assigns zero votes to the Sad Puppies in cases where the two slates were identical. (There's no way to tell them apart.) Looking at the categories where we can tell a difference, we see that the Sad Puppies averaged about 75 voters to the Rabid Puppies' 190. They were much less disciplined as well, with a 32% normalized standard deviation vs. the Rabid Puppies' 19%. These results should come as no surprise to anyone, but it's nice that the algorithm confirmed what many people had suspected: Without the Rabid Puppies, the Sad Puppies would never have swept any categories.

Click to Expand
(The light blue tip of the Best Pro Artist bar represents the split Rabid Puppy slate. There were zero Sad Puppy slate voters in that category.)

For 2016, the Sad Puppies claimed they were producing a recommendation list, not a slate, and they offered more than 5 choices in most categories. To allow for that, we adjusted the definition of slate voter to include anyone who voted solely from their recommendation list. Even with that change, the algorithm detected almost no one use used their list as a slate. Since that was their stated goal, congratulations are in order.

Again, this is not a measure of every voter who considered him/herself a Sad or Rabid Puppy. We're only estimating the ones who voted the slate, the whole slate, and nothing but the slate in each category. Someone who voted for a few items from the slate but either stopped early or else picked a few different choices counts as an organic voter by this measure--in that category, at least.

In particular, this means that we cannot tell whether all but 20 Sad Puppies simply switched to the Rabid Puppies this year. One clue might be in the votes for "Sad Puppies Bite Back" under "Best Related Work," where we estimate 35 slate votes and 60 organic votes. This is a work that likely would not attract votes from anyone but a Sad Puppy, and the total of 95 actually compares favorably with their numbers from last year.

Hostages

There was a great deal of concern about the "hostages," which were people who found themselves on the slates, asked to be removed, and were told "no." Some people who ended up on the finalist list chose to refuse the nomination rather than let people think they got there unfairly. This raises three questions:
  1. Of those slated nominees who accepted the nomination, which ones would have been finalists even without the slates?
  2. Did an actual Hugo Award go to anyone who owed his/her place on the ballot to a slate?
  3. Did anyone decline who would have ended up on the final ballot anyway?
For the first question, here's the list of all the hostages. We haven't tried to confirm that all of these were people who asked to be removed from the slates. They're simply nominees with slate support who would have been finalists without the support of the slates. 


CategoryNomineeRabidSad
Best NovelSeveneves
Best NovellaPenric's Demon
Best NovellaSlow Bullets
Best NoveletteFolding Beijing
Best Short StoryCat Pictures Please
Best Short StoryHungry Daughters of Starving Mothers
Best Related WorkYou're Never Weird on the Internet (Almost)
Best Graphic StorySaga Volume 5
Best Graphic StoryThe Sandman: Overture
Best Editor, Short FormJohn Joseph Adams
Best SemiprozineStrange Horizons
Best SemiprozineBeneath Ceaseless Skies
Best FanzineFile 770
Best FanzineBlack Gate
Best FancastTea and Jeopardy
Campbell AwardAndy Weir
Campbell AwardAlyssa Wong
Campbell AwardBecky Chambers

Ironically, some of the Sad Puppy hostages were pushed off the final ballot by the Rabid slate.

Abagail Larson (Best Professional Artist 2016) is the only person ever to win a Hugo Award after being nominated on the strength of a slate.

Except for Black Gate, no one who declined the nomination in 2016 would have made the list without assistance from the Slates, so good call. (Black Gate would not have made the list in 2015 without the slates, though.)

In 2015, "Goodnight Stars" would have made the finalist list without slate support, but it was withdrawn anyway. There were no other hostages in 2015.

There were two hostages in 2014, although it doesn't seem to have concerned anyone at the time.

Lazy Puppies

There are lots of interesting bits of information in the complete results for anyone who's really into it. For example, you'll see that we estimate that only 16 organic voters nominated Space Raptor Butt Invasion. Since no one really believes anyone would have nominated "Space Raptor" in the absence of the slates, that gives us an estimate of how many "slate-influenced voters" there were. In other words, We think there were only 16 people who nominated "Space Raptor" without also nominating the entire rest of the slate.

The organic deflator for "Space Raptor" is barely more than 1.0, which means most of those 16 nominated this one story and nothing else--in the "Best Short Story" category, anyway. In other words, they were "Lazy Puppies," who couldn't be bothered to nominate more than one or two items.

The Curious Case of Best Pro Artist 2016

On March 30, 2016, with just 24 hours before the deadline for nominations, Vox Day changed the Rabid Puppy Slate to replace Rowena Morrill with Larry Rostant as nominees for Best Professional Artist. Larry nevertheless got 258 votes, of which 240 can be attributed to the slate. Rowena got only 79. (Whatever mechanism he uses for his slating operation, it can turn on a dime!)

Since the algorithm assigned zero Sad Puppies to that category, we reran it with a special secondary slate representing the original Rabid slate with Rowena still in it. It still assigned 240 of Larry's votes to Rabid Slate #2, but it now assigned 78 (all but one) of Rowena's votes to Rabid Slate #1, for a total of 318 slate voters in that category. We have used that number throughout this article, and therefore the numbers for Best Professional Artist 2016 will not agree with the tables. Because it's worth comparing the two, we created a separate table for the 2016 Hugo Nominations Best Professional Artist. The most significant change is that it shows that Abagail Larson, the actual Hugo Award Winner, would not have been a finalist without support from the slate.

Predictions about the Future

We've established that there were between 300 and 400 hard-core slate voters in 2016, essentially all of them from the Rabid Puppies camp. How many should we expect to see in 2017?

This is a harder thing to estimate, since it depends on guessing how many people will pay for memberships, but we can look at how many appear to have voted on the final ballot. To do that, we start by looking at the Hugo 2016 Rabid Puppies ballot, which gave a recommended set of works and rankings. If we look at the categories where his #1-ranked work was from his slate and was not a hostage and then look to see how it fared in the round where No Award won (in the detailed Hugo 2016 stats), that'll gives us a fair idea. That is, we don't insist that Rabid Puppies voted his list in his order, but we do expect that they voted his #1 above No Award.

Best novelette is the first where this occurs, in round 4 when "What Price Humanity?" got 431 votes. 

In Best Short Story, "Space Raptor" got 659 votes vs. "Asymmetrical Warfare" in the round where No Award took position 2. One can attribute some of that to Chuck Tingle fans, though.

In Best Related Work, 412 people voted for "Between Light and Shadow" above No Award.

In Best Dramatic Presentation, Long Form, 457 people voted for "My Little Pony."

In Best Editor, Short Form, 766 people voted Jerry Pournelle somewhere above No Award. However, one expects a lot of people gave Jerry a vote out of nostalgia.

In Best Editor, Long Form, just 170 people voted Vox Day above No Award. This may be the most significant number of the bunch. It's very hard to understand why his followers would fail to vote for him.

In Best Pro Artist, 441 people voted Larry Elmore above No Award.

In Best Semiprozine, he urged people to vote No Award and nothing else. 312 people did so.

In Best Fancast, 166 people voted for Rageaholic.

In Best Fanwriter, 227 people voted for Jeffro Johnson.

In Best Fan Artist, 302 people voted for Christian Quinot.

At first glance, it looks as though some 400 of the Rabid Puppies were eligible to vote, but they did so half-heartedly. An alternate interpretation is that there were only 170 "true" Rabid Puppies who bothered to vote, but there are an additional 250 sympathizers who supported them here and there.

We can perform the same exercise with the Rabid Puppies' Hugo Recommendations 2015 and the 2015 detailed Hugo stats. Abbreviating, we get: Novella 556, Novelette 777, Short Story 525, Related Work 595, Graphic Story 810, Editor (S) 586, Pro Artist 877, Semiprozine 768, Fanzine 584, Fancast 550, Fan Writer 510, Campbell 538. An estimate of about 550 looks very reasonable.

This suggests that Rabid Puppy participation in the final voting dropped to half of what it was the previous year. This is consistent with the narrative that holds that after the puppies swept the nominations in 2015, a large number of them signed up to try to win the actual awards that summer. Those people were still eligible to nominate in 2016, but not to vote, and half of them did not bother to register for MidAmericon II. If that's really so, we'd expect to see the Rabid nominations fall in half next year.

An alternate explanation is that, knowing they couldn't win, fewer bothered to participate in the final vote, but they're still just as focused as ever on participating in the nomination round. Time will tell.

Complete Results

We have posted tables with all the data for every nominee. These are in order by organic votes, so they present a view of what the long lists might have looked like in the absence of slates. They're color-coded to make them easier to analyze.

Conclusion

The slates have been doubling in size every year, and they have greatly improved their discipline. At least 300 people strictly voted the Rabid Puppies slate in every single category. Although organic voters also increased their nomination rate, the puppies left them in the dust. 

Even so, for all that effort, the Puppies can only point to a single case where one of their candidates won an award that she would not have won without their help.

It remains to be seen whether the Puppies can repeat this level of nominations next year. Their voting rate on the final ballot suggests they'll drop back to their 2015 level. 

Appendix: Calculations

Algorithm

How do EPH numbers help us estimate the number of slate voters per category?

For each category, let's say there were S slate voters and they had R candidates on their slates. Then EPH will give their nominees S/R "points" each. If we look at the results of what the "E Pluribus Hugo" (EPH) algorithm would have produced had it been in effect, we can see how many points each nominee had at the point the algorithm eliminated it (or at the end when there was a list of 5). Comparing that to the actual contents of the slates, we can compute R for every item on the ballot. (Again, this comes from our definition of a slate voter as someone who votes the slate, the whole slate, and nothing but the slate.)

But not all the votes come from slate voters--not even for slated works. Instead, the total EPH points, P, will be
Where T is the total votes and Q is the Organic EPH deflator. That's just the amount that EPH scales down organic votes on this candidate. Of course we don't know either Q or S, but we can rearrange the equation to let us compute Q as a function of S.
Further, we can certainly compute Q for all the non-slate candidates: Q = T/because For organic candidates the actual EPH deflator and the organic one are the same. If we compute the actual EPH deflator for all candidates, we'll see that the slate candidates generally have larger values for Q, and that gives us the key to estimating S. For each category, we want to pick the value of S that makes all of the Q values "as similar as possible" to each other. In mathematical terms, we want to find the value of S that minimizes the standard deviation of Q in that category. (Strictly speaking, we used the variance of Q weighted by the number of organic votes.)

Because S is a small integer, it's easy to write a program that simply tries all the values, computes the resulting standard deviations, and picks the best one.

Adding the effect of a secondary slate, like the Sad Puppies vs. Rabid Puppies, is a bit more complicated, but the corresponding equations are these:
and
Where S₂ is the number of voters from the second slate and R₂ is their equivalent of R. (Number of secondary-slate candidates still available at the point EPH discarded the candidate.) Computationally, this is a good bit more expensive, since you have to evaluate every possible combination of S and S₂, but it still only takes a couple of seconds if you do it right.

Raw Data

Pulling together all this data was a real chore, involving copying from various Hugo documents and slate web pages, cleaning it up and integrating it with Perl scripts, and making manual edits when nothing else would do. As a public service, we are providing the raw data that we used for the algorithm described above. They are in the same "EPH order" as the original reports submitted to the business meeting.

2014 Hugo Nominations Raw Data

If anyone finds errors or omissions, we'd love to know about it. Note that these tables only contain nominees that were in the EPH reports presented at MidAmeriCon II. That means there is no information for the two Best Dramatic Presentation categories. Likewise, a couple of nominees from the bottom of the actual long list are missing because they weren't in the EPH top 15. Those were replaced with things with names like "Newly Listed Entry."

References

Official Slates

Rabid Puppies 3 Final Slate, 2015.
Hugo Recommendations 2015 (Rabid Puppy Final Ballot)

E Pluribus Hugo Analyses

Analysis for 2016.
E Pluribus Hugo slide presentation from Sasquan
A Proportional Voting System for Awards Nominations, by Jameson Quinn and Bruce Schneier, presents EPH formally in the context of voting systems in general.

Hugo Awards

2016 Hugo Awards and detailed statistics.

Other Estimates

2016 Hugos: Some Initial Stat Analysis, Brandon Kempner (Chaos Horizon), August 20, 2016.
Random Thought - 2016 "What Could Have Been" Hugo Finalists, Aaron Pound, (Dreaming About Other Worlds) ,September 20, 2016.
Biased Opinion - 2016 Hugo Awards Post Mortem, Aaron Pound, (Dreaming About Other Worlds), September 13, 2016.

1 comment (may contain spoilers):

  1. I love your analysis. The link for what the "E Pluribus Hugo" (EPH) algorithm would have produced is no longer working. I guess Midamericon removed it?

    ReplyDelete