Saturday, March 19, 2016

Rocket Stack Rank at Six Months

Rocket Stack Rank went live on September 19, 2015, so we're six months old today. We set out to create a web site that would make it easier for fans to find good short science fiction and fantasy (SFF) to read prior to making nominations for the 2016 Hugo Awards.

Toward that end, we did everything we could think of, culminating in our 2016 Hugo Awards recommendation page, which put all the information together in a convenient place, and the evidence shows that a modest number of users are starting to make use of it.

Traffic Statistics

The Google Analytics chart below gives some idea of how things have gone so far.

A few things jump out:
  • We achieved 95% of our goal of 10,000 unique visitors. (We're currently running at over 2,000/month.)
  • During December, A single, super-popular post, Health of Hard Science Fiction in 2015, got as much attention as a whole month of regular material.
  • People who visit the site tend to look at three or four pages before they leave.
  • Volume has been fairly level during the nominating season.
  • People are still getting to know the site; most visitors were first-time visitors. (Currently still 50%.)
From studying other analytics, we think there is a core group of about 500 people who regularly read the reviews and access the external content. It's not a bad start.

Original Goals

Our original goal was to read and review all the short fiction in the six major publications in 2015. We accomplished that and also included all the original fiction from ten anthologies.

We hoped that would amount to 50% coverage of the stories in the Locus Recommended Reading List, but it actually came to about 65%.

We set out to offer advice on where to buy copies of back issues of the big three print magazines. We ended up with detailed instructions for several different ways to get electronic copies of back issues, and we even discovered several (legal) ways to borrow back issues without having to buy them.

Beyond Reviews

Other Short-Fiction Reviewers

The weakest part of our original plan was rating the stories. We're both software engineers and long-time SFF readers, but it wasn't clear why anyone should take our reviews seriously. So we found eight other sources for reviews and cross-referenced the stories we read with the stories those other reviewers recommended. This let us produce much stronger recommended lists while giving credit to the other reviewers too. The results for the whole year are summarized on 2015 Year-To-Date page.

In the process, we honed our own reviewing skills and developed a set of criteria that we're very comfortable with.

Artist Nominations

We put a lot of work into our 2016 Professional Artists and 2016 Fan Artists pages, which not only make it easy to flip through hundreds of images but also include links to the artists' own sites. Both artist pages got over 200 visitors who actually spent enough time on the pages to have made lists of nominees. In 2015, only about 300 people made fan-artist nominations and only about 750 made pro-artist nominations, so we feel good that we might have made a positive difference here.

Editor Nominations

Our most controversial post of the past six months was 2016 Editors (Short Form). Of all the work we did, this one came the closest to being a "slate." Probably the most substantive complaint was that we only considered editors from the top six magazines plus the nine anthologies we'd read. 

For 2016, we've expanded the number of magazines we read, so we're hopeful that won't be as big an issue next year.

Campbell Nominations 

The easiest extra feature we added was a page to help people find new authors to nominate for the Campbells. We created the 2016 Campbell Award Eligible Writers page just by filtering our main list based on a couple of eligibility lists.

Other Recommendation Lists, Annotated

We noticed that a lot of other recommendation lists didn't make it easy to find the stories they were recommending. So we repeated the exercise we had done for the Campbells.
In all these cases, we did some extra reading to make the annotations more complete for stories that hadn't been in our original group of books and magazines.

No Comment

Comments are one place we didn't get the results we wanted. We received 331 comments, but almost half of them were written by us. We still hope the site can become a place where people discuss their favorite (or least-favorite) stories, but that hasn't quite happened yet.

Goals for the Next Six Months

  • We'd like to end the first year with about 25,000 unique visitors (so 15,000 more). 
  • We want to review all the stories from 11 top magazines plus 12 original anthologies.
  • We'd like to produce at least one more "hit" blog post like "Hard SF" was.
  • We want to cultivate a community of regular readers who visit and comment every month--not just during Hugo season. The new Discussions page does a better job than the Feed gadget in the sidebar at surfacing recent comments, and we've done some automation of our recommendations on Facebook posts and Twitter tweets, but there's room for improvement.


We think we're off to a good start. More important, we're both still enjoying it. We look forward to continuing this project in the years to come. We'd welcome feedback on what readers would like to see.

17 comments (may contain spoilers):

  1. I just wanted to comment that I love this site and find it very useful for fans like me, who have a hard time wading through all the available short fiction. I intend to be a return visitor for the forseeable future, and I'm looking forward to seeing how your reviewing expands in 2016! :)

    1. Thank you very much. So do you have suggestions?

    2. I'm not sure I have many suggestions beyond your stated goals--it would be great to see more magazines and anthologies covered. Also, I would be interested to read more reviews on standalone novellas (i.e. not published in magazines), since that's a length that seems to be taking off in e-books lately.

      Also, one random thing, it would be really useful if you could include in your magazine info pages the date of publication per issue (since that doesn't correspond to the month on the issue for some magazines).

    3. I'll look into adding the month/year of magazines with numbered issues, after Hugo Award nominations close March 31. I've got a bit more reading to do. :-)

    4. Regarding standalone novellas, we review the ones from (though sometimes a month or two after release). Here's the novella review link; it's also linked in the sidebar under Stories By Category and the Year-To-Date page.

      As for the date of publication, it turns out to be tricky due to different dates for the digital, print newsstand and print subscription issues. Initially I thought you wanted the month/year for numbered issues like Clarkesworld and Lightspeed. If you need it to determine Hugo Award eligibility, use the cover date, which is what's shown in RSR reviews.

    5. As far the the Hugos go, the cover date rules, but a lot of people don't seem to realize this.

      For example, the January 2016 issue of Analog came out in November 2015, but none of those stories is eligible for the Hugo this year; they're all treated as if they were published in 2016.

    6. Oh, I also didn't realize this. I was having some issues trying to figure out the actual publication dates of each magazine :). Thanks for the explanation!

  2. I made much use of your site in determining word length for various short fiction on my personal nominations lists, and for details of sources, etc. I also enjoyed reading your mini-reviews and comparing them to my impressions. I couldn't find everything on my list, but that makes sense, since you only did six mags for 2015 (still impressive given the late start).

    I also found some fiction by favorite authors which I had not come across, by searching under author names. I expect I will find more ways to use the site this year. I appreciate the work you two have done more than I can say.

    1. We appreciate the kind words. And it's nice to know that all the hard work we put into getting word counts does get used! :-)

  3. I've been lurking around the site for a while, trying to keep up with the major magazines this year and enjoying the mini-reviews.

    1. Oh, and if you are looking for suggestions maybe an index for finding all stories by magazine somewhere on the sidebar perhaps?

    2. Thanks for the suggestion. I've added a list in the sidebar with shortcuts to each magazine in the YTD By Magazine page which has the recommended stories for each magazine grouped by issue, as well as links to issue reviews at other sites like SFRevu, Tangent Online and Goodreads.

  4. Kudos on the excellent first half-year :D

    I've been using RSR a lot, it went on my Hugo ballot, and I'm hoping to put some effort into commenting more often.

    Here's one suggestion that may be useful: RSR is fantastic on breadth, but that same breadth might be making it hard to build up a critical mass of comment activity. With so many unique pages, discussion that takes place isn't very visible, and the site structure doesn't encourage discussion in any particular way.

    I've been quite pleased with my short story group, which is just a simple Facebook group - but by choosing one story to spotlight a couple of times a week, the active members keep coming back and participating (yay!). Something like that might work well for RSR too - a "Featured Story," focusing visitors' attention on one particular piece at any given time.

    A featured story might have a longer, more in-depth review; maybe have some discussion questions. (You'd want to pick stories that have potential for discussion beyond loved-it/hated-it/meh.) You might invite guest posts from other readers or reviewers; that'd be a great way to engage a wider audience and bring discussion in -- a guest one week would be more likely to come in as a commenter the next.
    Or, take a look at what Niall Harrison did when he ran a short story club - he announced the list of stories, and every week he collected reader and reviewer responses to form an initial post. Then discussion would bounce off all of those. Of course, members of the club had extra incentive to write something about the story in advance -- that'd give them the first word, as it were. It was a good thing, and worth emulating, or at least taking into consideration :)

    Free stories are always easier than non-free stories, but if you're concentrating on 6 major magazines, you could rotate between them, e.g. feature one story a week.

    All the best!

    1. That's a pretty good idea. We need to put some thought into exactly how to implement it. Possibly a pinned message on the home page directing people to the story of the week.

    2. Yay!

      Once you have it, you've got plenty of ways to direct people to it. Once-a-week-ish is infrequent enough to go on the non-reviews feeds, on the Twitter account, etc.

      If you make even a minor event of it, you can probably get tie-in attention from the author, the venue, fans.

      But then, you guys are awesome at figuring these things out. It's something I'd love to see; I hope it works out :)

    3. Those are all good ideas. Also, the story doesn't really have to be one from the current issue of some magazine; it would be okay to use older ones, I expect, although I think we'd want to stick with ones from 2016.

    4. Speaking as an individual with an internationally-shipped subscription to F&SF in print, whose delivery time is consistently erratic, I support this qualification!