Tuesday after lunch marks a time in the week that cannot be pinned down. It's not the beginning, end, middle or even expectation of anything special. But it is special in my little world, because that's when we have `journal club'.
Journal club is a uniquely science thing, part of this odd occupation and (I would contend) essential to its successful pursuit in the modern age. I most sincerely hope that it is part of your routine. If somehow you have let this ritual slip from your working week, it might be time to put it back for two reasons: (1) there are a lot of wonderful papers out there, more than perhaps ever before in the history of science, and it is very helpful to chat about why they're good; and (2) there are a lot of really terrible papers out there, more than perhaps ever before in the history of science, and it is very helpful to chat about why they're bad. The part that makes it awfully confusing is that many of the most awful papers are in those journals we often consider the best, and the part that is wonderfully confounding is that many of the most wonderful papers are in those journals we often consider, well, not awful, but not the best.
We should take a moment to see why this is, and to do that we might have to revisit the process of scientific publication and how this works. And how, despite it being unquestionably the best system we've ever tried for the dissemination of scientific information, we nevertheless let bad things happen to good papers and vice versa. Let's start with those journals we generally consider the terrific ones - you know, the glossy ones or the ones with really soft pages. How does it happen that anything ever gets published in these edifices to excellence, these monuments to marvels?
It starts with professional editors. These are individuals who have (in nearly every case) a rigorous scientific background and have chosen a career path wherein they contribute to science through selflessly (and I mean this without sarcasm) working to bring the best work to light in one of the best (that is, their) journals. They read huge stacks of papers (not only those that have been submitted but generally those being published in other journals of similar stature) and choose those areas, emerging ideas, and specific papers that should be focused upon. They have a remarkable breadth of knowledge and work extremely hard at what they do. They generally identify outstanding papers to be brought to light, following review by experts in the field whose opinions they personally trust. It is a system that we hold dear, because we hope we'll publish some of our own work in these journals.
So how can it be that bad papers appear in these journals? And do not doubt for a minute that they do. If you tend to take as gospel truth everything you read in publications with impact factors greater than, say, 20 (or for that matter any journal), I should let you go ahead and do it, because you're not going to be effectively competing with my lab. Worse (or perhaps better, for me), if you are the sort who simply looks over the abstracts of high-impact-type papers and files away the info for future reference, you are going to crash and burn. Yes, there is a huge amount of information out there, and it seems as though nobody can process all of it, but you simply cannot take the information in an abstract as true and succeed as a scientist - not for long anyway. But why do these papers get in in the first place?
Again, it starts with the editors, who screen the papers and are looking, primarily, for something new, because new things keep the journal's impact factor high (because more people read new things than old things). However, and here's the catch, most things that are new and exciting are simply wrong (which, of course, is why nobody noticed them before). At this point, though, the editor has done his/her job: identified something new and exciting that, if correct, would be suitable for publication in a high-impact journal. Now it's up to the reviewers.
And here is where the dice are rolled. Three reviewers who have the time, the willingness and, I hope, the expertise have to agree to even look at it. Most of the time, these erstwhile reviewers will not perform the actual operation of reading and evaluation but instead rely on the services of a graduate student or other trainee who, sad to say, might not have properly learnt to evaluate a paper through participation in a journal club. Yes, the conclusions of the paper are certainly exciting but only if the experiments support those conclusions. And the decision as to whether or not this might be the case is going to be up to you if the paper gets published. And it may get published, even if the experiments are shoddy, the conclusions are not supported by the data or there are unconsidered alternatives to those results. Ultimately, it is you who decides whether these conclusions will be internalized into your world view, the zeitgeist that dictates how you will spend your time in the analysis of the world through the scientific process.
During a recent journal club in which we eviscerated a particularly poor paper published in a journal of usually exceptional quality (we could not determine whether a single conclusion of the paper was even likely to be correct), one of the participants in our discussion asked me, “If papers like this appear in journals like this, how can we aspire to publish in these same journals? If we can't believe papers in this journal, what can we believe?”
Believe what you decide is probably true, of course. And save only your best work for those journals where, if you are lucky enough to get it published, your work will be studied and discussed in journal clubs all over the world. See to it that at the end of the discussion your paper has faired better than most, and has been integrated into the world views of other scientists and labs. Make sure it is work you can be proud of.
But not all work is published in the `best' journals, of course. And there is much to consider in why we must extend our journal club discussions to papers published in all sorts of journals. We've still got a lot of work to do before next Tuesday, after lunch, rolls around.
