Q&A

Contents

What is the purpose of Rocket Stack Rank?

We want to make it easier for people to nominate short works of speculative fiction (short stories, novelettes, and novellas) for the Hugo Awards. We saw that there were two big obstacles to doing this: the first is that there is such breadth of new material that most people can only read a subset. The second is that even if someone recommended a story, most people have no way to get hold of it unless it was offered online for free.

We read original stories in eleven magazinesAnalog Science Fiction and Fact, Apex, Asimov's Science Fiction, Beneath Ceaseless Skies (BCS), Clarkesworld, Interzone, Lightspeed, Strange Horizons, The Magazine of Fantasy & Science Fiction (F&SF), Tor.comand Uncanny, and we rate them on a scale of one to five, with the intention that most stories will get a score of three, and that few will get either one or five. Someone with very little time could simply read our five-star stories and choose from those. Someone with more time could read the fours and fives.

Additionally, we have documented different ways to get the back issues of magazines so people can read stories from earlier in the year.

How do I get copies of the magazines for stories?

Each of our reviews includes a link to access/borrow/purchase the issue that the story appeared in, Our Reference page describes all the different ways to get any of the magazines we're reading. In broad terms, there are four strategies, although you can mix and match:
  1. The Kindle strategy. If you have a Kindle device, go to Amazon.com and subscribe to Analog, Apex, Asimov's, Clarkesworld, Lightspeed, F&SF, and Uncanny. Subscribe to BCS and Interzone with Weightless Books, and opt to have your subscription delivered to your Kindle. Then simply read the stories as they come to your device. Use the Desktop strategy (below) for Strange Horizons and Tor.com and for back issues.

    This is the most comfortable and convenient approach, and it gives the most financial support to the magazines. (And they desperately need it.)
  2. The Desktop strategy. If you have a Windows, Macintosh, or Linux desktop or laptop with a browser (e.g. Internet Explorer, Firefox, or Chrome), read new and old issues of Apex, BCS, Clarkesworld, Lightspeed, Strange Horizons, Tor.com, and Uncanny on the desktop for free. Read old issues of Analog and F&SF for free via your public library's website. Buy copies of Asimov's for Magzter and read it with Magzter's web reader in the browser. This is the cheapest approach. Note: At present, we know of no way to read Interzone on the desktop.
  3. The Tablet strategy. If you have an Android or iPad tablet, install the free Kindle, Kobo, and Google Newsstand apps. Subscribe to the magazines via Kindle, but purchase back issues from Google Newsstand or Kobo, depending on which format works best for your device. 
  4. The Print strategy. Subscribe to Analog, Asimov's, Interzone, and F&SF in print form. Order back issues from their web sites or buy them on eBay. Check your public library's website to see if it subscribes to these magazines. Use the Desktop strategy for the online magazines.
 Again, this is just a broad summary. Look at our Reference page for details.

How is Rocket Stack Rank different from other reviewers?

We're trying to build a recommendation system that people can use to select stories to read, get access to those stories, and come back and share their thoughts after reading. It is very difficult to write a review for a short story that doesn't spoil it. For that purpose, we divide our reviews into three parts: the rating, the "blurb", and the "mini-review." The rating tells you whether we think the story is worth reading or not. The blurb is a very short general description that doesn't spoil the story. The mini-review is displayed upside down until you click on it, and you shouldn't look until after reading the story. It explains how we arrived at the rating we did and serves as a jumping-off point for further discussion.

As far as we can tell, no other reviewers are doing anything like this. They write reviews that contain spoilers, and they are seldom very critical of any story. They're interesting to read after you've already read a story, of course, but they're not much help if you only want to find the top 10% best stories to read.

Another distinction is that our reviews are focused on the plot sophistication and character development of the stories. We care how well they're written and (for the very best stories) how emotionally engaging they are. We're not concerned about issues like novelty or inclusiveness. Plenty of other reviewers already do that, and we include their recommendations so readers can mix and match if they want to.

Why should we take your ratings seriously?

You don't need to! You can look at other people's reviews or recommendations and then use our site to help find the magazines so you can read them. Add your comments to tell us why you think we're wrong about one story or another.

That said, we're trying to apply consistent standards to all the stories we read, and we do compare our reviews with those of other reviewers. See Recommended By below.

What do the Different Ratings Mean?

We use a five-star rating system, like many others, but we want five-star scores to be very rare; most stories should get three stars. Most of the rest should get two or four. One and five-star stories should be rare.

Not Rated is for stories that are not self-contained (a serial) or may be well-written but are barely science fiction or fantasy. E.g. David Gerrold's "The Martian Child" and Rachel Swirsky's "If You Were a Dinosaur, My Love." Those are both excellent, moving stories, but, in our view, neither has enough speculative content to qualify for the genre.

One star (needs improvement) identifies stories that have "Writer 101" problems. Things that shouldn't have left the slush pile. POV errors, editorializing, info dumps, as-you-know-bob, Mary-Sue protagonists, nonstop purple prose--that sort of thing. You'd think nothing in a professional publication would ever have those problems, but it does happen.

Two stars (not recommended) are for stories that for whatever reason keep kicking you out of the narrative. E.g. the story is in Pennsylvania, but the capital is Boise. Or the spaceship is in a spiral orbit (which is impossible). Or a robot's subroutines bounce when it falls. Likewise stories that don't seem to have an ending, and those that appear to have no plot at all. (With exceptions as described below.)

One and two-star stories tend to be painful to read. They leave you feeling exhausted, and not in a good way.

Three stars (average). Most stories should fall into this bucket. Nothing wrong with it--nothing great about it. You should be able to read through it without losing suspension of disbelief, but it's possible that on later reflection you'll think "wait, that wouldn't have worked!" Simple, linear stories also go into this bucket, as do stories with too many "loose ends." E.g. a story where the reader keeps asking "why is he/she telling me this?" or one that ends with the reader asking "what about X? what about Y?" likely has a lot of loose ends.

A flawed story that is nevertheless gripping will (as of mid 2016) get three stars instead of two. The underlying rule is that no matter what problems a story has, it will never get fewer than three stars if it was actually pleasant to read. Vignettes (stories that show off a world but in which nothing happens) will generally get three stars on these grounds. (Previously a plotless story could never get more than two stars.)

This is by far the broadest category.

Four stars (recommended). Starting in 2016, there are two ways a story can get four stars: plot sophistication or memorability. The hallmark of a four-star story is that it gets better the more you think about it. Reading it twice is usually rewarding either because you see how numerous details of the story worked together to support the conclusion or because it's moving emotionally. A four-star story makes you think and it sticks with you. (Before 2016 a moving, memorable story with a simple plot would have only earned three stars.)

Five stars (award-worthy). In broad terms, these are well-crafted stories that both have sophisticated plots and are memorable/moving. These are the stories that suck you in. You feel the characters' pain, you share their joy. A five-star story often moves you to tears. You don't just feel you witnessed something special--you were there. It's typically difficult to read another story immediately after reading a five-story story; you have to finish processing it. And it can linger with you for days.

Each level operates as a filter. A story can be emotionally powerful yet still only get one or two stars because of defects in the writing. That said, we don't apply the rules pedantically. One or two blemishes won't disqualify a story from a high rating. We always give the story the benefit of the doubt.

Having rules makes it easier to talk about why we gave a story the rating we did. We almost always know how we'll rate a story as soon as we've finished reading it, but the rules help us organize our thoughts when it comes time to explain the ranking. In a few cases, we explain why a story is great despite the fact that it breaks some rules, and we've revised the rules a couple of times based on great stories that didn't fit the old rules.

I Saw That You Changed a Review Score. Why Did You Do That?

This is a "Stack Rank," which means that the scores are determined by combining results from several sources. At certain points during the year, we'll reexamine the stories in the highest and lowest buckets and reconsider their rankings. A one-star story that got recommendations from other reviewers certainly merits further thought. Likewise, if we seem to have too many 5-star stories, it's worth looking back over the ones that no one but us liked to see if we were overenthusiastic about a story for one reason or another. A couple of times, readers have commented that we missed the point of a story, causing us to reread and rerank the story.

That said, we don't actually have a quota for the number of 5-star stories, so we won't drop good stories just to meet an arbitrary number. Once the Hugo nominations open, we won't make any further changes in the scores for that year.

How is Recommended by determined?

We survey a few long-time, prolific, short fiction reviewers who read more than 500 print and online stories a year to contrast their recommendations with our own. The :number suffix represents our attempt to map those recommendations onto our one-to-five-star system, as follows:
We include this information to give readers more choices in selecting which stories to read--we don't consider ourselves the final authority by any means.

How do you define the different categories?

Rocket Stack Rank targets readers who want nominate short fiction for the Hugo Awards. Accordingly, we use the same definitions they do:
  • Novel: At least 40,000 words
  • Novella: At least 17,500 but fewer than 40,000 words
  • Novelette: At least 7,500 but fewer than 17,500 words
  • Short Story: Fewer than 7,500 words
When the word count differs from what a magazine reports, we use the category from the magazine, but we still report the word count. This may happen because the Amazon Kindle (from which we get most of our word counts) appears to slightly undercount the words.

If a work is marketed as a novella, we'll still consider reviewing it here provided it has fewer than 50,000 words.

How is this different from a Slate?

In our view, a slate is a single set of five or fewer stories per category which people are urged to vote for without reading them. Our goal is simply to help people read more stories so more people can make nominations. Slates are about not reading; we're about more reading.

Second, we aren't producing an absolute ranked list of stories. (We won't say "this story was #1, and this one was #2.) We're creating a partial ranking. That really means all we're doing is giving each story from 1 to 5 stars and then we're sorting by number of stars. When we list the 5-star stories, they'll be sorted alphabetically, and likewise for the other groupings.

Finally, we produce more than one list. We have our own rankings, we give lists based on other reviewers, we do lists by magazine, and we even have a composite list that combines everything together.

How is the Time Estimate calculated?

According to this Forbes article (and Wikipedia), the average adult reads prose at 300 words per minute (wpm), so we take the story's word count and divide it by that rate to get the time estimate. The article also points out other averages such as college students at 450 wpm, "high level execs" at 575 wpm, and college professors at 675 wpm. If your reading speed is at a higher rate, you should adjust the time estimate down by a third, a half, or more. We provide the time estimate as an upper bound for our readers, who we think are above average. :-)

What are the "mini-reviews?"

A mini-review is a short explanation of why we gave a story the rating we did. It usually has one paragraph about what we liked about the story (that is, why we gave it this high a rating) and another paragraph about what we didn't like (why we didn't give it a higher rating). Think of it as the jumping-off point for a real discussion. Feel free to use the comment section to express your own thoughts. (Note: comments can and do contain spoilers.)

Are you a fanzine? Could you be nominated for an award?

The Hugo Award FAQ does say that blogs count, and the official definition requires just one "issue" per year, whereas we publish a new "issue" on the 15th of every month when we publish the monthly ratings table, so, yes, we do seem to qualify technically. We would accept an award, if we thought we were fairly nominated for it, but we don't plan to campaign for one. In 2016, we received 66 nominations, which placed us ninth.

Our cute little rocket is public-domain clip art which we found on the WPClipart site with these terms: "These images are public domain (PD), and that means they can be used and edited for whatever purpose you wish, personal or commercial. No attribution or linking is required." But we wanted to credit them anyway.

How Do I Submit Magazines/Stories for You to Review?

We're currently reading eleven magazines, about ten anthologies per year, all of Tor.com's stand-alone novellas, and selected stories from other sources that other reviewers have recommended. This doesn't leave us a lot of time to look at stories or publications from sources unknown to us. However, we'll at least consider anything people send to us.

We want to be reviewing the stories that are most likely to be considered as nominees for the major Science Fiction and Fantasy Awards, particularly the Hugo Award, but since we have limited time and energy, we have to select a subset of all the thousands of stories that appear annually. Toward that end we use the following guidelines:
  • We only review original fiction. We won't review reprints from previous years under any circumstances, nor will we review nonfiction. (A translation isn't the same thing as a reprint; first appearance in English is what counts.) This is because we're focused on works that are still eligible for awards. Sometimes a story is reprinted in the same year it was originally published. (E.g. originally printed in January and then reprinted in September of the same year.) That kind of reprints is fine.
  • We mainly review Science Fiction and Fantasy--not Horror. We'll review a horror story that appears in one of the magazines we regularly review, but we won't review a magazine or anthology that's primarily or exclusively horror. We definitely won't review a stand-alone horror story, and we won't review anything that has no speculative element in it.
  • We only review works that are available in electronic form: either as eBooks or as web pages. We won't review anything that is only offered to the public as print on paper.
  • We only review short fiction, so we won't review anything over 50,000 words under any circumstances.
  • We try not to review anything under 2,000 words. Again, we'll review original flash fiction that appears in one of the magazines we regularly review, but we won't consider a magazine or anthology that's primarily flash, nor will we consider stand-alone flash stories. The reason for this is that our rating system places a lot of weight on plot sophistication and character development, and most flash fiction just isn't long enough to do well by either criterion.
  • We mainly review stories that appear in SFWA-Qualifying Markets. We made an exception for Interzone simply because so many high-quality stories were turning up there that we were buying 1/3 of the issues anyway. Failing that, we'd favor semiprozines with regular publication histories and which pay something close to the SFWA-qualifying rate (currently 6¢/word). We're very unlikely to review a story from a fanzine or a blog unless it comes highly recommended by someone we trust.
  • We closely follow what other reviewers are recommending, and we'll consider stories, anthologies, and publications that they rate highly. We treat major recommendation lists like the Locus Reading List and the major year's-best anthologies as special kinds of reviewers. When possible, we try to review entire anthologies or publication issues rather than single stories, even when it was the recommendation of a single story that led us to it.
  • We pay for almost everything we review; we do not solicit free copies. We don't turn them down, though.
If you want us to look at something, your best bet is to suggest single stories and tell us the genre, the length, the per-word rate, and where it's available to the public. If we see enough good stories from a specific magazine, we'll add it to our list. To contact us, send e-mail to greghull@rocketstackrank.com. 

Why do you give negative reviews?

There is a school of thought that holds that you shouldn't say anything if you can't say something nice. Even if we just said nothing instead of giving 1 and 2-star reviews, it would still be obvious what we thought about those stories. Also, we do want people to use Rocket Stack Rank as a place to discuss stories; that means we have to list them--and the best discussions are likely to be around stories that some people loved and other people hated. Finally, one can argue that ratings below 3 stars are a criticism of the magazine, not the author. That info will be useful when it comes time to make nominations for best editor/best semi-prozine.

That said, we do have concerns about hurting people's feelings--even though we've been assured that authors have a very high tolerance for criticism and rejection. We try never to criticize an author personally nor a magazine; our critiques are entirely directed at the content of the stories.

What is your comment policy?

Rocket Stack Rank encourages people to leave comments discussing stories, articles, or reviews. We appreciate getting corrections for errors, and we occasionally even change the rating of a story based on feedback. Obviously-bad comments, such as spam, personal attacks, off-topic rants, etc., are subject to deletion without notice.

How do you make money from this?

We don't. This is our retirement project, and we're doing it strictly for fun. The site is not monetized in any way whatsoever: there are no fees, no ads, no affiliate links, and no appeals for donations, nor do we think there ever will be.

and
(Updated December 29, 2016)

7 comments (may contain spoilers):

  1. Is there a way to send you short fiction (such as anthologies and novellas) for review? Thanks.

    ReplyDelete
    Replies
    1. You can send suggestions to rocketstackrank.com if you like. The username is greghull. There's no guarantee we'll actually read any given submission, of course, but we're definitely interested in sources beyond the 11 magazines we follow.

      Delete
  2. Greg,

    My name is EXO Books--my pen name at least. It's also the name of the publishing company my wife and I created to put out my science fiction.

    You have made it very difficult to contact you guys! I am hoping to send you our debut novella, The Last Day of Captain Lincoln. How can I do that? Happy to offer one of our beautifully illustrated little hardcovers.

    Best wishes,
    EXO Books

    ReplyDelete
    Replies
    1. If you look up at the answer I gave Scott, it actually does tell you how to contact us via e-mail.

      We only review things that can be read online--preferably as e-books (MOBI format), so if the book only exists in print, then you'll have to wait until there's an e-book version.

      We don't usually read self-published material either, unless it already has some buzz around it or is by a famous author. There's just too much of it.

      Delete
    2. Greg,

      I've been poking around this site for a while now and still can't make sense of your reply to Scott (which you refer to in your answer to me). Maybe I'm just daft. You say "[y]ou can send suggestions to rocketstackrank.com if you like," but that's not a valid email address. You provide a user name but I can't seem to find a place to use it on this site!

      I would love to send you my debut novella but I can't seem to wrap my head around what you're trying to explain. Please help!

      Thanks,
      EXO Books

      Delete
  3. Regarding "All Your Cites I Will Burn".
    Pros: It's funny (see below). If you live in the south, I expect you'll find parts of it hilarious. The pokes at Research Triangle Park, the degradation of I-85, the gated communities of mini-mansions are only too real. In the end, it is an acknowledgement of mass extinction events, and the occurrence of evolution. Evolution is not always pretty. And it's not always to safe to discuss it. The Meteor Gods are illustrative of fear of change, but very much congruent with a southern country (don't hate me, I'm southern), that lives on KFC, Krispy Cream and Cook Out Milkshakes - and hefts around the landscape gobbling up whatever is edible. I think it is about someone realizing that life is to be lived regardless of Meteor Gods. One never knows where we're headed in the long run.

    Cons: The Meteor Gods are just enough beyond the pale to decrease my ability to suspend disbelief. A bit post-apocalyptic and therefore depressing for my tastes, but I get the theme.

    ReplyDelete
    Replies
    1. Thanks for posting! I grew up in the South myself (Chattanooga), so I know what you're talking about.

      Did you mean to post this with the original review for All Your Cities I Will Burn? You've somehow posted your comment on the Q&A page.

      Delete