Summary
Progress in biomedical research depends in part on being able to build on the findings of other researchers – and thereby on being able to apply others’ methods to your own research. However, most of us have struggled to understand how to repeat or adapt another researcher’s study because of minimal or missing details in the Methods section of a published paper. In expensive and complex experiments involving animal models, clear descriptions of the methods are particularly important. In this and the accompanying Editorial in this issue, we discuss how crucial the Methods section is to the integrity and utility of a biomedical research paper, and encourage researchers working with animal models to follow the recently released ARRIVE guidelines when preparing their studies for publication.
There’s madness in our Methods
When you publish, you get to claim scientific priority for a discovery. In return, you share what you know in sufficient detail so that others can reproduce and build on that discovery. What is surprising is that many papers fail to report experimental procedures in sufficient detail for this to happen. I do not think it is intentional in most cases; rather, I think that we tend to use other published papers as examples of what to report and, facing word limits at many journals, often the Methods are the first place we cut corners. We do this even with the knowledge that compromising detail in this section of the paper might be detrimental to readers.
“When papers don’t provide experimental procedures in sufficient detail…another group may have to start from scratch, wasting valuable time and resources. This is unfortunate in any case, but especially so – some might even say unethical – when the procedures involve experimental animals”
During my PhD in Peter Walter’s lab, part of my first major project was to figure out how to separate and purify the proteins in signal recognition particle (SRP). It took about a year of testing different kinds of columns and loading and elution conditions, but I finally got it right, and I published the experimental procedure in my first research paper in 1985 (Siegel and Walter, 1985).
A couple of years after I started my postdoc, a technician in Peter’s lab called to tell me she was going to purify some SRP proteins, and asked me whether my published experimental procedure was complete and reproducible. I told her I thought it was, but I was terrified that I had left out some crucial step, and that the purification in her hands would fail – which would mean that the paper I had published was flawed because, without the purification, the experiments using the purified proteins would be impossible to reproduce. When she called a few days later to announce her success, I was relieved.
In a research paper that you write, the Methods usually comprise some experimental procedures that you have personally fine-tuned or even designed, and others that were established by others or that you might consider standard. Those that are new and yours alone to describe in detail are probably the ones reported most precisely in your research articles. Then there are scores of other experimental procedures that have been handed down from colleagues or in lab notebooks and which you consider standard. Rather than describing these in detail in your article, you might reference a previous paper that you know used the method, even if you’re not sure the method is actually detailed in that paper – and if it isn’t, you expect it will reference the paper in which it was first described. And then there are the experiments performed at a core facility at your institute, which you trust have been done according to some standard procedure – and you may or may not obtain and report those precise details in your paper. You look to other papers to see what details they have reported, and you include what they include, leave out what they leave out.
On the flip side, almost all of us have tried to repeat the experimental procedures described in a published research paper, only to find ourselves frustrated by the lack of detail and the need to look up references to other papers that also don’t provide the procedure we’re after. So, we start on a wild goose chase, reading the Methods sections of several generations of referenced papers eventually to find the goose, or, alternatively, contacting the author to find out what they really did – and hope that they know.
When papers don’t provide experimental procedures in sufficient detail to know precisely how the experiments were performed at a level of detail sufficient to reproduce them, another group may have to start from scratch, wasting valuable time and resources. This is unfortunate in any case, but especially so – some might even say unethical – when the procedures involve experimental animals.
How, then, can we improve the situation? When I teach manuscript writing, I urge my students to provide methods like a ‘cookbook’, replete with the detail necessary for anyone to reproduce the experiments. That’s great advice, but – as a procedure for manuscript preparation – it too is faulty, because it lacks the detail to really guide our students. What we really need are detailed guidelines – checklists – for describing every type of experiment.
Thankfully, these guidelines are starting to appear across many research areas. Perhaps the most well known of these are the CONSORT guidelines for clinical trial reporting (http://www.consort-statement.org/consort-statement/), which enable both clear reporting of the clinical trial itself and also systematic reviews of related trials that may be informative in a way the individual trials were not. More recently, in 2010, a study group comprising animal researchers, journal editors, statisticians and research funders convened to develop guidelines for the reporting of research involving experimental animals [see accompanying Editorial in this issue (Percie du Sert, 2011) and http://www.nc3rs.org.uk/ARRIVE]. DMM is pleased to endorse these guidelines and to recommend them to our authors and reviewers. Their recommendations for writing the main text of the paper, from Title through Discussion, are for the most part very sensible, but I especially value the guidelines for describing methods. I urge our authors to use them as a checklist when preparing manuscripts for submission, and to read the accompanying Editorial from the Programme Director of the center that provides these guidelines (Percie du Sert, 2011).
We welcome the development of guidelines for research reporting, and will continue to endorse and to publish guidelines that we find relevant to our authors. As one example, in 2010, we published a special article called ‘Standard operating procedures for describing and performing metabolic tests of glucose homeostasis in mice’ by Ayala et al. for the Mouse Metabolic Phenotyping Center Consortium (Ayala et al., 2010). We encourage you to participate in guideline development in your research area and to publish those guidelines in DMM, where they will be immediately freely available upon publication for everyone to read and to use.
