Having already written about the need to independently test results, I’m pleased to see a news article in Nature that highlights the following initiative by Science Exchange to replicate high-profile papers:
Scientific publishers are backing an initiative to encourage authors of high-profile research papers to get their results replicated by independent labs. Validation studies will earn authors a certificate and a second publication, and will save other researchers from basing their work on faulty results.
It’s done through a service called the Reproducibility Initiative: here, you submit your study, and the Science Exchange will draw upon their 1000+ expert providers for validation. We also hear from John Ioannidis (he of most research findings are false fame):
John Ioannidis, an epidemiologist at Stanford University in California, is on the initiative’s scientific advisory board. He expects only authors of high-profile papers to submit their work to extra scrutiny, and says that the project could help the scientific community to recognize experimental design flaws. “A pilot like this could tell us what we could do better,” he adds.
Besides companies like Scientific Exchange, there’s also a huge opportunity here for publishers to offer this service as part of the process in publishing a scientific paper. One of the messages I’ve tried to get across in this blog is the need for academic publishers to think outside the box if they are going to survive. As it currently stands, publishers primarily act as a middleman between authors and peer review, so why can’t they go one step further and offer the opportunity for other labs to independently test the results?
A big problem is providing incentives: why should laboratories confirm someone else’s work? It’s nice to believe in this charitable contingent of academia that would simply undertake such a task for the good of advancing knowledge. Sadly, the likely scenario is that publishers pay labs to do this (as part of the service fee charged to authors) or they find some other means of getting these labs to do it for free (reputation model of being mentioned in a paper… it’s something to put on your CV, right?). The Science Exchange initiative, for instance, does this on a fee-for-service basis. Still, even if we get past this hurdle, then there is still the problem of whether or not an author wants to have their work scrutinised and re-tested in such a manner. Again, it would be nice if this was just part of the culture of doing science, but I’m reasonably confident there would be a lot of private (and public) protesting in some quarters.
Plenty of tropes have been levelled at academic publishing. But throwing your hands up and saying the writing is on the wall isn’t something I particularly buy into for the following reason: we don’t know what roles academic publishers are going to play in the future. I agree with some commentators that journals can no longer rely on the business as usual mentality — the ecology is changing and only those who adapt (or are lucky) are going to ensure long-term survival and success. If deployed correctly, the resources and clout available to academic publishers will make them valuable service providers in academia. Offering new services, as mentioned above, will ensure publishers remain part of the academic landscape, even if the big players will have different names to Nature and Elsevier.
Baker, Monya (2012). Independent labs to verify high-profile papers Nature News DOI: 10.1038/nature.2012.11176