I'd like to thank you for a year of splendid wisdom and wit, which I found extremely enjoyable and made the tribulations of scientific struggle a little more bearable.
But now, about your cunning plan to establish a better system to avoid fraud and reduce fudging (J. Cell Sci.119, 5007-5009) – it's a splendid idea, but despite eBay, unlikely to work for science. This is partly because you're asking people to rate technical and intellectual prowess rather than mere commercial integrity. In your example of the bright, fortuitous and ambitious scientist, it is implicit that these individuals will become the dominant group in the area, as the others are removed by attrition (unsuccessful grant and tenure applications). The `training' that you propose to administer would hinder the speed of accession of these stars, and thus would be selected against. And the consequences of a field dominated by these Wunderkinder? A big trendy literature, topical but inwardly focused (and bearing some resemblance to what we have now).
A case in point might be recent advances in reverse non-radioactive dot blots, also known as arrays: the technology du jour. These have been vigorously applied to cancer research to find out how tumours grow and how we might stop them. It is true that some discoveries have been made (possibly random, as these datasets contain huge numbers!). However, interest in `tumour stem cells' has returned. This idea had fallen by the wayside but, if it turns out to be true, all of that megalicious profiling is somewhere between worthless and mildly interesting (because such cells would make up only a tiny fraction of the tumour and their `signature' would be obliterated by their descendents).
Your system would require a `revisionist' approach, leading to revised opinions of formerly hot papers published in trendy journals. Neither the authors nor the journals would be keen to pursue this, although the wisdom accumulated would benefit those entering the field. Alternatively, the journals might publish helpful editorials along the lines of The Emperor's New Clothes. OK, neither is going to happen, but at least I can revel in the satisfaction of `I told you so'.
Caledonian Caveman (name and address supplied).
What a splendid letter! Much to ponder here. About all sorts of things: arrays, tumor stem cells, naked emperors and hot papers. And extra points for using “megalicious” in a sentence!
The problem, as I see it, is this. How can someone who does not have access tothe `inner sanctum' get an inkling of which findings are generally accepted ina field and which are widely considered flawed? My suggestion was a forum anyone can access, but as you note this is fraught with unpleasant difficulties. Ideally, impartial arbiters would pre-evaluate such critiques – but who would monitor the arbiters?
Which brings us to your other point. Will trendy journals leap to correct the literature or just pretend that it simply isn't of interest any more? What if a tree falls in the forest and many people hear it, but it turns out not to be a tree, but an elephant? Should those who reported hearing the timber be obligated to communicate the elephant problem? And if they do (and some, but not all journals, do) is it enough to do so quietly on-line, while the casual literature-searcher is still thinking that it was, indeed, a vegetable phenomenon?
The problem stems from a misunderstanding. We, as scientists, seem to think that journals (even high-impact journals) exist to disseminate information, and therefore should be obliged to report on changes to this information as more knowledge accumulates. But that's our mistake – journals are businesses, and their business is to sell copy. If there is a new finding, it is published because we (often via the proxy of our institutions) buy the journals in order to read new things. We tacitly (or not so tacitly) support this system, because we reap rewards when we publish in the best journals. But have you ever tried to publish a paper that simply points out that the results in another paper are not reproducible, or otherwise flawed? First, it is rare that such work is generally viewed as `interesting.' And if you try to publish it anyway, reviewers will demand that you explain why the authors of the work you are challenging got the result they did.
So you won't publish it, but you'll show your friends, and they'll share with you their skepticism of the original work as well. And soon you and your friends will be an `inner sanctum', and you'll snicker when you overhear me, in my ignorance, cite the original work as interesting and valid. Pity.
I do want to find a way to get the word out in a forum that isn't fraught with the politics and in-fighting that comes with criticizing scientific observations and conclusions. Something like the Royal Society of prior centuries. No, wait, that was all politics and in-fighting.
Yup. You told me so.