The report, “Peer Review in Scientific Publications,” released last week by the U.K. House of Commons Science and Technology Committee, reiterates much of what we already know:
- That peer and editorial review is important for maintaining the integrity of the scientific literature
- That the process of peer review is not consistent across all journals
- That pre-publication review may be supplemented — although not replaced — by post-publication review
- That publishers need to continue experimenting with other models for review and dissemination
- That editors and senior academics need to educate new scholars on how to provide quality reviews
- That granting and promotion committees should not rely upon a single metric (e.g., the impact factor) in order to evaluate the merits of a paper
The report also repeats a recommendation that the United Kingdom set up an independent office dealing with cases of suspected scientific misconduct, similar to the United States’ Office of Research Integrity.
What is new in this report is a recommendation that researchers provide access to their raw data to editors and reviewers at the submission stage, and then to the public after publication:
The presumption must be that, unless there is a strong reason otherwise, data should be fully disclosed and made publicly available. In line with this principle, where possible, data associated with all publicly funded research should be made widely and freely available.
While the premise of open access to data for the purpose of verification is deeply grounded within the ethos of science, such free sharing of data is rarely followed unconditionally in practice.
A 2009 study of authors who published in PLoS journals illustrated that even an explicit policy requiring the sharing of data as a precondition of publication goes largely ignored: only 10% of authors were willing to share their dataset with the inquiring researcher.
Scientists do share data, although they do so carefully, sparingly, and on their own terms. This doesn’t mean that scientists are hiding evidence of misconduct; it means only that making all of one’s data public is onerous, comes with few rewards, and distracts from producing new science.
There are other reasons why primary data do not flow freely from researchers. For one, editors and reviewers often don’t want to see them. Last year, the Journal of Neuroscience suspended the practice of publishing supplemental data because it was burdening reviewers, overwhelming authors, and slowing the publication process. For similar reasons this year, the Journal of Experimental Medicine followed suit, permitting only “essential” supplementary data to be published with articles.
Reacting to the recommendation that all scientists provide public access to their data, Tracey Brown from the non-profit commented for Science:
It is not clear from the Committee’s report what the problem is that would be addressed from raw data publication nor the other costs and effects of demanding it.
While the House of Commons’ report is filled with rich details for anyone interested in the practice of peer review — its strengths and limitations — I find the recommendations that stem from these details somewhat contradictory.
On one hand, the House committee understands that the practice of peer review is highly variable, that it reflects the different values and needs of individual communities, and that experimentation should be encouraged at the grassroots level.
On the other hand, the committee is also willing to make grand claims about the system as a whole and propose system-wide changes that appear to ignore how peer review is actually practiced.
When the integrity of a process is challenged, transparency is often cited as a solution. It is not evident, however, that open data mandates would improve the integrity of the peer review process, although it may provide a public appearance of it. More likely, it will greatly increase the costs of conducting good science, slow the process of discovery, and put UK scientists at a disadvantage over their international peers.
Before public officials push to legislate such requirements, it would be useful to understand what they’ll be sacrificing in return.