Ouch. Just got the rejection of our lovely paper that we worked so hard on (on which we worked so hard, thank you Ms. Rosenberg, my sixth grade grammar teacher), and which the reviewers took apart in no time (okay, about a month and a half, but who's counting? Oh, right, I did). They had a few valid points, but generally missed the big point, but that's how it goes. This week we will carefully go over the comments, swear repeatedly, and go back to the drawing board. That is, we'll start the long process of revision, submission to a new journal, review, evisceration, revision (hopefully), re-review, and possibly another return to the drawing board. (I just looked it up, ‘back to the drawing board’, most likely comes from the caption a circa 1940s cartoon by Peter Amo in the New Yorker in which an engineer throws away a blueprint while a plane crashes. The New Yorker is one of those magazines I leaf through to look at the cartoons, which are often great. Like the writer sitting at a typewriter – it's an old cartoon – surrounded by various dogs and another person advising him, “write about dogs”. This one was from George Price. What were we talking about? Oh right, rejection). It's the worst.
This isn't a good day.
When we review a paper, it isn't our job to think up more experiments to do. It isn't our job to identify questions that the authors haven't asked. It isn't even our job to help the authors make the work ‘more interesting’.
Most of us, and by ‘us’ I mean biomedical research scientists (not ‘insectivores’, although if you, like me, are an insectivore, please ping me. We can have ‘tea’, sometime), were taught to read papers in school. The assignment, in general, was ‘read this paper and tell me what's wrong with it’. And when we do journal clubs we often do the same (I hope you do a journal club – this is a very valuable exercise. Journal clubs should not be how you get your information, that's what reading is for. But they are very good ways to discuss scientific rigor and help to set our standards for good science. Good journal clubs are a great way to evaluate your own work in a conversational setting, as in “If that isn't a good experiment, what would they think of mine?”). We are taught to tear papers apart. You don't get an ‘A’ for saying, ‘I thought this was really good’. And when we are asked to review papers, we do as we were taught.
But that isn't what reviewing is. When you review a paper, you are effectively standing between the authors and those who will eventually read the paper (maybe me). Every week, month or year of additional work demanded is time lost to the scientific community, and therefore time lost to scientific progress. Satisfying your personal curiosity is not the goal here.
So, I think we need some guidance in how to review a paper. I don't know who to ask about this, and therefore, I'm going to just make it up. Here, for your amusement, then, is ‘Mole's Guide to Reviewing a Paper’.
Are the conclusions of potential interest to the readers of the journal? This is, perhaps, the toughest one to break down into a ‘do it this way’ sort of approach. For the most part, we follow the advice of Supreme Court Justice Potter Stewart in his written opinion in the case of Jacobellis vs Ohio in 1964: “I know it when I see it.” If you are being asked to review a paper, you are presumably an expert, familiar with the field, and you know if the conclusions would potentially impact your field. If you don't, then you shouldn't be reviewing. There are many reasons that the answer may be ‘no’. The conclusions may have already been widely accepted, based on strong experimental results by others. The advance in knowledge may be very small (reviewers tend to use the term ‘incremental’). Or the conclusions might simply not be interesting to those who read the journal (for example, if the journal is widely read by people outside this particular field, would any such readers care about this particular conclusion?). The last one is tough, because often that last bit is not really up to you, but you are allowed to weigh in. But if the conclusions are potentially interesting, you should say so.
Do the data presented support the conclusions? This is where the ‘meat’ of your review should be. Where, specifically, do the data fall short of supporting the conclusion? If, for example, the authors conclude that a particular thing is important in a disease, and all of the work is in vitro, then you can point out that the data do not support the conclusion. Or a vital control may be missing. You can also point out an alternative conclusion that could be reached using the same results (in congregate – you can't just pick on one result if other experiments already rule out your notion). Or maybe they did not show that the effects they describe are robust (by this I mean reproducible in the context of their conclusions – if they conclude that ‘cells’ do something, but only show it in one cell type, you can ask about how general it may be). Sometimes (even often), any of these could lead to a different conclusion, and here, again, you are allowed to re-evaluate #1. But here is what you should not do: demand that the authors perform an experiment that has not been done and tell them what result they must obtain. That isn't how we do science. Tell them why you do not feel that the experimental results do not support their conclusion. You can suggest the type of experiment that might help, but you cannot tell them what their results must be. They may decide to do something better. (Warning, detour!). Once, not so many years ago (in Mole years) a reviewer asked us to do an experiment that current knowledge predicted would give a specific result. Try as we might, we could not obtain the desired outcome. Fortunately, we were able to publish the paper anyway, and a few years later, we published another paper explaining why this experiment didn't work. Things were more complicated than we thought. (Sadly, another paper under review at the time of our first paper did provide data to support the expected result, which we now know was wrong – presumably the authors wanted to satisfy the same reviewer and get their paper in and provided results from a single experiment that sort of looked okay – not a good way to do science). (End of detour, but I hope I made my point: biology is complicated and demanding a result from an experiment that has not been done can end badly for everyone). Remember, you stand between the paper and those who might want to get something useful out of it.
What would make the paper more interesting? Yes, you are allowed to suggest such things. But if the conclusions are interesting and supported by the data, anything you suggest In this regard should be considered optional. It could well be that the authors will agree and do what you suggest. But again (it is worth repeating) the delay in getting the results out to the community have to be worth it. And by ‘worth it’ I mean not only time, but also money – science is expensive.
Don't find rabbit holes to navigate. Ever since Charles Dodson had Alice Lidell follow a rabbit to an adventure underground, we have had the expression to ‘go down the rabbit hole’, where things get curioser and curioser. Rabbit holes in a paper can be considered new questions that arise from the results, and it is very tempting to ask, “well, how does that work?” The answers, as we know, can sometimes take months (or years) and represent an advance in knowledge. And they inevitably lead to more questions (“okay, then how does that work?”). Not every paper has to have all the answers to be valuable. It can be fine to say (probably in the discussion) “we don't know”. If it is genuinely interesting, we will probably find out someday. Indeed, it often happens that in the rush to explain “everything” in a paper, we get a lot wrong – and if the interesting conclusions can be reached without knowing everything, then demanding answers now does not help anyone in the long run.
Please, when you review a paper, do not simply create a laundry list of more experiments. If the conclusions are simply not supported by the data, have at it (that is, tell them why). But satisfying your curiosity is not the goal here. Nor is burrowing ever deeper into ‘how it all works’. Interesting conclusions, well supported by experimental data, are all we want as readers. Don't obstruct – facilitate. Help the authors give us something we want to read, and we'll all thank you. (And don't be fooled when authors say, “We thank the reviewer for this thoughtful question, and in answering it, we now provide what is essentially a second paper in the supplemental figures”).
By the way, getting a bad review is not ‘the worst’ thing in the world. It just feels that way for a little while. Indeed, this is why one of Mole's Rules (someday I'll count these, but for now just assume that there are a few of these. Which should not be confused with ‘Mole Rules!’ which someone graffiti'd on the wall outside my office window. Not me, of course. What were we talking about – oh right, Mole's Rule). It is this: Never respond to the editor of a journal within 24 h of receiving a negative review. I know, I know. “Mole”, you say (I'm listening), “I am always very careful to write a balanced, friendly, and nuanced response as soon as I can upon receiving a bad review”. But you only think that it is balanced and friendly. Your adrenaline is up, you are angry, your autonomic response has triggered your sympathetic nervous system into fight or flight mode (and you are not going to fly, no way). So instead of saying “This is the sort of Philistine pig-ignorance I've come to expect from you non-creative garbage”, you just carefully say, “You excrement”, in a nicely balanced, friendly way. (Thank you, John Cleese). It never leads to anything good.
The worst, in my opinion, might be having your lab shut down for 2 years due to a raging pandemic, but let's see how that turns out. Me, I'm going to have to meet the Molet and decide where to try next. Hopefully you'll get it to review. Be nice, please! And I will do the same.
