Wrong. Wrong, wrong, wrong. What a waste of time. Excuse me, but I'm going to throw a hissie-fit. Okay, I'm back, but I'm in a really bad mood.

You see, we just wasted some time and money doing experiments based on a paper that was very exciting, and published in a journal with nice, soft pages. And it all turned to, um, excrement. Because the basic observations we had trusted to be true were not. Oh well, we'll move on, but it's annoying.

We all know this problem. There is a great deal in the published wisdom that is simply not right and, if we go through the trouble of checking it out carefully, we spend time proving that it isn't right and we find ourselves in the unenviable position of not having anything to do with the results of our efforts.

I can hear you – publish it, Mole! We need to know what isn't right in the literature, and if possible, why it isn't right. But more often than not, the answer is boring, or worse, the original result is simply not reproducible and we dont know why. Or even worse, we have our suspicions why. How do we accentuate the negative?

Before we get into this further, I want to digress a bit (it calms me down). There is a push to ‘clean’ the literature, and to determine to what extent the published canon is wrong. This comes in two ‘flavors.’ One of these is led by industry, resulting in widely publicized press reports of the failure to reproduce ‘landmark findings.’ The second is in the form of more academic efforts to broadly assess reproducibility. I want to have a look at each of these before getting into the ‘negative results’ thing. Why? I don't know, I'm just feeling grumpy.

With regard to industry efforts, again these come in two flavors. In the first, scientists who work in industry examine claims that a particular molecule or process plays fundamental roles in a variety of diseases. This makes sense, because if they can convince themselves that this is true, especially for a disease with a ‘large market share,’ they may decide to invest a lot of money in targeting the molecule or process in hopes that it will be useful. I have no problem with this at all. This is similar to what we do in academic research; deciding if a particular avenue of research is worth our efforts. And if we decide it is not, giving voice to our skepticism has value. It is not necessarily definitive; there may be interesting reasons for why a particular set of experiments didn't work in our hands, but if we decide not to pursue it for good reasons, letting others know this can be helpful in their own decisions to work on this or not. And of course, this usefulness will depend a lot on exactly how we reached our decision not to pursue a course of study (what convinces me may not convince you). That's all good scientific discourse.

But I do have a problem with the second form that this effort in industry takes. This is where a project is described in which the researchers report that they failed to reproduce a large swath of ‘landmark findings,’ without telling us what they mean by ‘reproduce’ or ‘landmark.’ If you have ever sailed long distances (I have, which can be described as very long periods of sleep deprivation and monotony interspersed with sheer terror – pretty great really), you know that a landmark is a navigational tool – it is something that you can steer by without error. For a finding to be a landmark, it has to have consistent value in guiding our research. So by definition, I doubt that landmark findings are not things that are reliable. But that does not necessarily mean that everyone can ‘reproduce’ the exact finding. [Warning, another detour ahead.] Long ago, I was deposed as an expert witness for the defense of a university that was being sued by a government agency for scientific fraud. I spent a long hour or so arguing with the plaintiff's attorney about data produced in several in vitro cellular experiments that showed consistent effects; the attorney asserted that since the actual values in different experiments (done on different days) were different, the results were not reproducible. It took me a long time to realize that our definitions of ‘reproducible’ were completely different (the case was dropped). [End of detour!] So, we are told that many ‘landmark findings’ were not ‘reproducible,’ without knowing what was meant by these terms. Failing to reproduce a result might mean, ‘we got the same effect, with different values,’ ‘we could get the effect, but not with sufficient consistency to have value in a drug screen,’ or ‘we used different reagents and cells and conditions and the experiment didn't work.’ We have no idea. But, oddly, many scientists I met rejoiced at this assertion and took the conclusions at face value. These are people who would not believe their own results until rigorously tested and retested. I don't get it. Maybe it is schadenfreude, the derivation of pleasure from someone else's misfortune (what a great word; is there a word for the deriving displeasure from someone else's fortune? Sure. Envy.)

Given the tone of such reports, and the lack of actual data (or methodology), I have to wonder about the intention of this sort of thing. In the first case (reporting skepticism of a conclusion based on reported experimental results), this is clearly meant to inform discussion among scientists. But in the second case, what is the goal? It seems to be an effort to undermine public faith in the scientific enterprise; more, it seems to suggest that ‘you can trust us, you can't trust them.’ Is the hope that public support for academic research will divert to industrial support? (Why not? At least in the country in which I live, massive amounts of government support are provided to some industries that would do just fine without it. I'm sure that isn't true in your country, though.) Just saying.

But what about the problem of reproducibility and the reporting of negative results? We'll get there. Along with the academic efforts to address these issues. But right now, I'm going to have some ‘tea,’ cheer up, and get over it. Hey, it will be a little vacation. You take one too. Enjoy! Then we're all going to get back to work.