Your guide to Open Access publishing and Open Science

Independent Confirmation of Results and Academic Publishers: A Potential Opportunity?

Author: 4 Comments Share:

Having already written about the need to independently test results, I’m pleased to see a news article in Nature that highlights the following initiative by Science Exchange to replicate high-profile papers:

Scientific publishers are backing an initiative to encourage authors of high-profile research papers to get their results replicated by independent labs. Validation studies will earn authors a certificate and a second publication, and will save other researchers from basing their work on faulty results.

It’s done through a service called the Reproducibility Initiativehere, you submit your study, and the Science Exchange will draw upon their 1000+ expert providers for validation. We also hear from John Ioannidis (he of most research findings are false fame):

John Ioannidis, an epidemiologist at Stanford University in California, is on the initiative’s scientific advisory board. He expects only authors of high-profile papers to submit their work to extra scrutiny, and says that the project could help the scientific community to recognize experimental design flaws. “A pilot like this could tell us what we could do better,” he adds.

Besides companies like Scientific Exchange, there’s also a huge opportunity here for publishers to offer this service as part of the process in publishing a scientific paper. One of the messages I’ve tried to get across in this blog is the need for academic publishers to think outside the box if they are going to survive. As it currently stands, publishers primarily act as a middleman between authors and peer review, so why can’t they go one step further and offer the opportunity for other labs to independently test the results?

A big problem is providing incentives: why should laboratories confirm someone else’s work? It’s nice to believe in this charitable contingent of academia that would simply undertake such a task for the good of advancing knowledge. Sadly, the likely scenario is that publishers pay labs to do this (as part of the service fee charged to authors) or they find some other means of getting these labs to do it for free (reputation model of being mentioned in a paper… it’s something to put on your CV, right?). The Science Exchange initiative, for instance, does this on a fee-for-service basis. Still, even if we get past this hurdle, then there is still the problem of whether or not an author wants to have their work scrutinised and re-tested in such a manner. Again, it would be nice if this was just part of the culture of doing science, but I’m reasonably confident there would be a lot of private (and public) protesting in some quarters.

Plenty of tropes have been levelled at academic publishing. But throwing your hands up and saying the writing is on the wall isn’t something I particularly buy into for the following reason: we don’t know what roles academic publishers are going to play in the future. I agree with some commentators that journals can no longer rely on the business as usual mentality — the ecology is changing and only those who adapt (or are lucky) are going to ensure long-term survival and success. If deployed correctly, the resources and clout available to academic publishers will make them valuable service providers in academia. Offering new services, as mentioned above, will ensure publishers remain part of the academic landscape, even if the big players will have different names to Nature and Elsevier.

Reference
Baker, Monya (2012). Independent labs to verify high-profile papers Nature News DOI: 10.1038/nature.2012.11176

Previous Article

Open and Shut: #OpenAccess #OpenData #OpenScience Around the web

Next Article

Open and Shut: #OpenAccess #OpenScience #OpenData around the web

You may also like

4 Comments

  1. This is only possible in certain fields where there are standard methodologies for performing experiments (unlike many engineering research for example). Furthermore, good peer reviewers could do a good enough job with much less trouble, if peer review communication could be made quicker, more open, both ways, and more rewarding (i.e. paid and/or attributed). This latter way would make things much more effective and sustainable than the author’s suggestion.

    1. Good point on your first line: this is only applicable to experiments (although I was under no illusion that it wasn’t).

      But where did I suggest that peer review communication wouldn’t be made quicker and more open? Also, how does making peer review more open allow for a more effective and sustainable environment for independently testing results? Independently testing results and peer review are two distinct things. All I was suggesting is that publishers should perhaps provide independent testing as a service in an analogous manner to how they currently provide peer review.

      1. Engineers also perform experiments, I did not mean that we don’t, and most are difficult to replicate. That is why I would prefer to build upon the work already done rather than carbon copy it. As for peer review, the first step to open the process must be to make it not blind and slap the name of the reviewer on the paper, thus encouraging constructive reviewing.

  2. Dear James,

    just came accross this post and would like to share a little update on how things are moving on, now that the “irreproducibility crisis” gains even more high level attention. Besides measuring (and complaining about) the irreproducibility of studies, Science Exchange, us, and some more players in a project funded by the NIH have moved ahead to provide partial solutions to this multidimensional problem. As antibodies-online.com’s key competence lies in product data, we all are starting to tackle this from the research materials side (I am not saying, that this is the only aspect, but it is one part of the reproducibility equation). We currently proceed with two approaches:

    1. Make sure high quality reagents are used in a reproducible manner (this meaning, that a proven protocol exists on a per product, per application basis) (see more on http://www.antibodies-online.com/independent_validation/ or http://reproducibilityinitiative.org/#/projects).

    2. Make sure, research materials used can be uniquely and systematically identified (see more on http://www.antibodies-online.com/resource-identification-initiative/ or http://www.force11.org/Resource_identification_initiative)

    Just wanted to keep you up to date. Feel free to just let this go through as a comment or contact me, in case you would like to get more information on this.

    Andreas Kessell
    Co-Founder http://www.antibodies-online.com

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.