“It's too bad she won't live,” says Gaff, “but then again, who does?” Oh man, dark dark dark. I've just watched Blade Runner for the first time (this year, that is – umpteenth time overall, of course), and it gives me chills on so many levels. Mostly good. And yeah, I like both versions. If you haven't seen it (really?) it's about hunting down rogue replicants – synthetic humans – except of course it isn't about that at all. But yeah, there's a lot of rain. Someone once said that the director, Ridley Scott, never shot a dry sidewalk.
But that has nothing to do with what I wanted to talk about. I wanted to talk about the reproducibility of published findings and something that has been going on for a few years but is getting worse. And it's not what you think. Stick with me (if you do, I'll throw in more Blade Runner lines. No, wait, if you do, I won't throw in more Blade Runner lines). There's a storm brewing.
As you may know, not everything that's published is reproducible. Maybe a lot of what's published. If you're thinking most, well, you're not alone (but I'm not going to agree with you). For the past few years, there have been flags raised, and then a growing concern, that we simply cannot replicate our findings. This concern is sometimes bordering on hysteria – what if we're just wasting our time? “We are!” you cry (some of you – yeah, I can hear you, Leon). We have to do something about this! Maybe we should set up facilities that seek to replicate findings, and if we can't, delete them before they do damage! I'm not kidding about this: there is a serious (okay, serious to me) proposal that applications for government-supported research grants not be considered until all preliminary findings are reproduced, ensuring that time, and money, not be squandered on things that don't actually work. Let's fix it, because certainly people who are paid to check things will get it right, and won't be swayed by false leads. And if the work can't be replicated (replicant, get it?), we can discard it and focus only on what actually works. Even in Blade Runner, the scanning electron microscope in the outdoor market worked. Then again, it was a pretty dystopic world (really, you haven't seen it yet? Come on – stop reading this and go watch it.)
Calm down. I mean chill. Panic will get us somewhere, but where it will get us is no place good.
Replicating results is something we do to do this thing we call research. We search and then we re-search. Throughout the history of science we have noted those findings that work and those that don't work, and yeah, there are always more that don't work. But with the rise of the enterprise and the ever growing publication list (23 million at last count, and rising fast), we can't try to reproduce everything and we're getting worried that lots of, um, bad stuff (and bull stuff) is slipping past us. We can't keep up. We have to do something! Well, someone has to do something – we don't have time.
The rising alarm is sometimes traced to a publication in 2005 by John P. A. Ioannidis: “Why most published research findings are false.” Actually, this paper followed a few others (some by the same author) and focuses on genetic associations with diseases and outcomes. It urges better statistical evaluation and study design, and provides many useful suggestions. But the concerns go way beyond such correlative studies: in a recent commentary, a researcher from the biotech industry pointed out that cancer preclinical studies that form the basis of further drug discovery efforts are plagued by irreproducibility. They apparently tried to reproduce findings in 53 ‘landmark’ papers in cancer biology and could only confirm six of them. They didn't say what these landmark observations were, but really? Six? That's 47 replicants! That's a huge number, even by Blade Runner standards (I know, I'm freaking out a bit too here, Leon!).
Again, they have made a number of useful suggestions, including developing approaches to the reporting of negative data and creating repositories for cell lines that can be used by researchers around the world. The latter is a really great idea, and fortunately already exists (google ATCC) but sure, this effort can be improved as well.
But I've got to wonder about these 57 ‘landmark’ papers. What makes them landmarks? As near as I can figure, to be a landmark, a thing either has to be a feature of the terrain by which you can navigate, or a turning point (or, I guess, both). I reckon by ‘landmark’ paper we mean the latter. But to be a turning point in a field, the work, by definition, must be useful, and to be useful, it has to have been not only reproduced, but extended into new territory. Because it was a turning point! Something in the work must be reproducible, or else we simply would not have been able to go down that road. So maybe the claim that only eleven percent of ‘landmark’ discoveries are reproducible is not reproducible? The folks who did this suggested that hundreds of papers had been based on these publications, but really, are these papers stepping stones on the new paths of a new viewpoint, or meanderings that happened to cite an earlier finding? What makes these landmark papers actually landmarks? And is anyone actually navigating by these beacons?
I suspect that these ‘landmarks’ were published in places that we might consider important, and as such, might get a nod in the discussion of another paper (hence, citation, hence, ‘landmark’). But what if, indeed, the observation published in one of these landmark papers have actually led to important advances, despite their ‘replicant’ status? Would we be better off having not gone down this path? I maintain that the basis for our panic is unfounded. If the work is useful, genuinely useful, is it even relevant if the original experiment does not give the iron-clad results we expect? (To be fair, the authors admitted that there could be technically challenging aspects to the work that might make replication less achievable with relatively limited effort.) Absence of proof is not proof of absence, and if you were to leave it all to my bench skills, I'm not sure I can replicate the experiment showing that things fall down rather than up.
“Wait a minute, Mole! There's a gaping hole in your logic here.” (Thanks, Leon.) Yes, there is. The problem is that even if folks can reproduce the basic findings and drive them forward (making the basic findings actual landmarks in the field), that doesn't mean that they can be translated into actual, useful treatments for disease in people. And really, that's why the public is shelling out hard-earned tax semolians for biomedical research instead of using them for other important things, like building really big statues. [FYI, a semolian is a unit of currency, not to be confused with a simolian (of the Sims' world), that is vital to the economy of cartoon cultures, such as mine. If you've never heard of a semolian, you have to watch Who Framed Roger Rabbit, right after you finish with Blade Runner. Besides being equally great, it will also cheer you up. I'm talking to you, Leon.]
Sorry, I got a bit side tracked (I can stop thinking, “shave-and-a-haircut, two-bits”). So let's see where we are: there is a growing concern, sometimes bordering on hysteria, shared by many scientists, that the literature is filling up with results that cannot be reproduced (replicants), and this is not only diluting the research effort directed at producing real health benefits, but may actually be undermining our ability to do so. There is an implication here that at least some of these replicants are the result of lying, cheating, or at the very least, selective vision due to bias (intentional or not). And we have to do something about it. Part of the rising hysteria in the scientific community is the realization that this is making its way into the public forum, and this could directly impact the entire enterprise – if the public lose all faith that biomedical research can provide cures for what ails us, then the stream of semolians will dry up. (You don't think so? Where have you been the last five years?) If we don't do something about this, someone will, and that someone may well have control of the funding.
Hey, this is the Mole you're talking to here. Of course I have some suggestions. But before we get to them, we have a lot more to consider. Stick with me. Meanwhile, you've got some movies to watch. So relax.
And pass the popcorn.