The Locus Reading List and the Hugo Awards

The annual Locus Recommended Reading List heavily influences the Hugo Awards, but there have been suggestions that it has biases against certain publishers, authors, and even types of story. Most of that criticism has focused on novels, so this article takes a look at the statistics and tries to see if there are any obvious patterns that suggest anything of that sort for the short fiction categories.

The Locus Reading List

Every January, Locus Magazine recommends over 100 novel-length works and between 100 and 150 short stories, novelettes, and novellas from the previous year. These stories are selected by a group of editors and reviewers, many of whom have been reading hundreds (or thousands) of SFF stories every year for a decade or more. Locus publishes the list in time to help fans choose quality stories to nominate for the Hugos.

For the purpose of this article, we'll use "Locus Reading List" and "Locus list" to mean the short fiction works (short stories, novelettes, and novellas) from the Locus Recommended Reading List.

As you can see from the chart below, the Locus editors propose roughly equal numbers of short stories and novelettes and relatively few novellas, and the mix hasn't changed much over time.

On average, the Locus list contains 132 short fiction works comprising 1,500,000 words, or about 15 novels worth of text. The typical reader would need over 80 hours to read the recommended short fiction, plus over 500 hours to read the 100+ recommended novel-length works, so it still takes a very dedicated person to actually read the entire Locus Recommended Reading List.

Where do the stories come from?

In a previous post, we talked about which SF magazines contributed the most Hugo and Nebula finalists over the past fifteen years. Applying the same analysis to the Locus list, we get this chart:

Over time, we can pretty much divide the list into roughly equal thirds: one third from the major magazines, one third from the minor magazines, and one third from anthologies/collections and "singles" (stories sold singly, aka "chapbooks.")

A few other changes over time seem to be clear:

• The three professional print magazines used to account for up to half of the Locus short fiction recommendations, but of late they have declined to about 13%. 
• The big three online magazines are now the source for as many stories as the big three print magazines. Others have commented on this decline.
• Minor magazines have declined in importance, but they are far more prominent here than they are in Hugo or Nebula nominations. (See our previous post,)
• Anthologies, collections, and singles used to account for about a fifth of Locus nominations, but lately they've grown to between one-third and one-half.
• The decline in Analog that we saw in the Hugo and Nebula awards is not as evident here.

Whatever else, the Locus Reading List does seem to be giving the smaller magazines a fair shake.

How much does the Locus Reading List Influence the Hugos?

It dominates them.

As the chart below shows, over the past 15 years, 79% of all short-fiction Hugo winners were part of the Locus list, and 68% of all Hugo finalists (aka Hugo Nominees) came from the list.

The chart below shows all Hugo Winners and Finalists, separating the ones that were part of the Locus list from those that were not.

In this chart "Winner" means "Hugo Winner that was not on the Locus Reading List" and "Finalist" means "Hugo Finalist that was not on the Locus Reading List."

Note the difference between 2014 and 2015. In 2014, a fairly typical year, two of the three winners were on the Locus list and six of the eleven finalists that were not winners were also on the list. By contrast, owing to the influence of the Slates, in 2015 none of the finalists in any of the three categories came from the Locus list.

To get an independent data point, we compared the Locus Reading List to the Nebula Awards. While the Locus list is produced by SFF editors, and the Hugo Awards are voted on by SFF fans, the Nebula Awards are given by SFF authors.

Note: Analysis starts in 2002 because Locus Reading List data only goes back to 2001 (online) but Nebula Awards had a 2-year eligibility window until 2010.

In general, the Locus list is somewhat less-correlated with Nebula finalists, but it is still a very strong predictor. If nothing else, this suggests that the Locus list really does contain the lion's share of all the quality stories produced in a single year.

Next we looked at the Locus Reading List influence on the Nebulas across time.

Note: 2011 has 4 Nebula winners due to a tie in the short story category.

Over the past fifteen years, the Locus list seems to have had roughly the same influence year after year. Strictly speaking, we can't tell if correlation implies influence: it's possible that the Locus list contains the best stories, so naturally the Nebulas will overlap with it--even if writers don't read the Locus list.

It appears that during the three-month Nebula nomination period, writers could nominate stories, see how each nominated story was doing, and could change their votes up to the end of that period. That's vastly different from how the Hugos work, and that gives authors a lot more ability to influence each other during the process. You'd expect that to reduce the direct impact of the Locus list (since writers have their own, much more interesting list) but there's no visible difference in the chart above. That suggests that the Locus list overlaps with the Nebulas simply because they're both lists of good stories.

Are some authors underrepresented?

What makes a significant difference?

If an author had a lot of Hugo and Nebula-nominated stories, but those stories didn't appear in the Locus lists, that would suggest that Locus had a bias either against the person or against the type of story. From the third chart (above), we know that a randomly-chosen Hugo finalist has a 68% of also being in the Locus reading list (hence a 32% chance of not being in the list). The chance that neither of two finalists were in the list is (0.32)² or 10%, and the chance that none of three finalists were in the reading lists is (0.32)³ or 3%. So we set our threshold for statistical significance at three.

Looking at the Locus lists from 2001-2015, we found 27 authors who had three or more Hugo finalists in the short-fiction categories. Of those, only five were underrepresented in the Locus list in a statistically significant way.

Two of them, Brad R. Torgersen (Hugo nominated 3 times between 2012 and 2014) and John C. Wright (Hugo nominated 4 times in 2015), openly used slate voting to get their nominations.

That leaves just three authors as plausible targets of discrimination. The question is whether they are similar politically or in the types of stories they write.

Mike Resnick: Hugo nominated 13 times between 2001 and 2012 with one win.

Mike Resnick writes science fiction. Wikipedia describes his writing as combining a love of Fable and Legend with a love of Africa. He has said he is not a liberal and that he has never asked anyone to vote for one of his stories, although he had no objection to slates in principle.

Michael A. Burstein: Hugo nominated 7 times between 2001 and 2006.

Michael A. Burstein wrote fairly hard, "dry," SF, mostly for Analog.  For his politics we know he has considered running for office as a Democrat. We can find no evidence one way or the other that he campaigned for awards for his stories.

Seanan McGuire (aka Mira Grant): Hugo nominated 4 times between 2012 and 2013.

Seanan McGuire writes zombie stories and other urban fantasy. Her politics aren’t obvious from her web site, but she posted a few anti-puppies things in 2015. Likewise, we can't find anything suggesting she campaigned for her stories to get awards.

If Locus really were discriminating against different authors or types of stories, we'd expect to see more people in this situation, and we'd expect them to be similar to one another. But that's not what we see.

Was it the quality of the stories?

Across all five of the above authors, there were 32 stories that were Hugo-nominated despite not being on the Locus list. Of those 32, one won the Hugo and six were Nebula-nominated. Since the average Hugo-nominated story has a 20% chance of winning the Hugo and a 39% chance of being nominated for the Nebula, we can use a binomial distribution to compute the odds that this occurred at random.

That tells us there is an 0.7% chance that 32 Hugo-nominated stories would win 1 or 0 times, and there is a 1.4% chance that 32 such stories would be Nebula-nominated 6 times or fewer. Both of these are statistically significant.

In other words, the statistics are consistent with these being lower-quality stories by the standards of Locus, the Hugos, and the Nebulas--all three. That doesn't mean they were bad stories--just that they weren't good enough to get awards.

As to how they got nominated in the first place, we know that two of the authors used explicit slates. It's possible that the other three benefited from "accidental slates," which, loosely, means that they might have had some over-enthusiastic fans who nominated works that weren't really good enough and which got on the ballot due to a weakness in the Hugo nominating process.

Conclusion

We cannot speak for its treatment of novels, but as far as short fiction goes, the Locus Recommended Reading List really does seem like an honest attempt to make it easier for people to find great stories. If it has an ax to grind, we can't find it. Its major problem seems to be that it just includes too much material for most folks to read within the Hugo nomination period.