This post, which was co-authored by Alice Meadows and Karin Wulf, is the first in a series to celebrate this year’s Peer Review Week, which they have also co-edited
The theme of this year’s Peer Review Week is “transparency.” Among the many planned global activities will be blog posts, both here on The Scholarly Kitchen and elsewhere, webinars and other events, and of course sessions at the Peer Review Congress on aspects of transparency, as well as a special Peer Review Week panel, Under the Microscope: Transparency in Peer Review.
To start things off on the Kitchen we wanted to go big, to be transparent about the values and general principles of peer review. The foundation of modern scholarly practice — across disciplines and rooted in the Enlightenment — is the process of testing assertions, followed by critical exchange among experts, and then more testing. But what is asserted, how it’s asserted, and who constitutes the community of expertise has always been up for grabs. Distortions of evidence, actual conspiracies and conspiracy theories, and the attractiveness of some theories over others have historically been rooted in factors including new media forms. In the nineteenth century, it was cheap print; in the twentieth century television and in the twenty-first century it is digital media that makes evaluating information and its deployment so challenging.
In the mainstream news, here at The Scholarly Kitchen, and on other blogs, there have been lots of hot — and some not so hot — takes in recent months on the challenge of information profusion, manipulated material, and distorted narratives. When we start to consider what constitutes reliable information, and how we know what we know — from image and video manipulation to social media bots to widely differential news emphases to retracted journal publications — the spectrum is vast. Yet if we return to the basic impetus for enlightenment inquiry, we might find that it is not certainty we seek (or even want), but confidence in the methods and process of discernment, both of which are deliberately lacking in fake news.
Peer review is a mechanism for critical discernment. It’s a vital element of how knowledge, in all its forms, is created through evidence and argument. The framework of alternative facts (in)famously defined by Kellyanne Conway as “additional facts and alternative information,” suggests that knowledge is a matter of opinion. Alternative facts, however, cannot withstand scrutiny, whereas knowledge actually thrives on it. Peer review, at its best, is one of the most rigorous forms of scrutiny available to us, and is thus key to the production of knowledge.
Of course, neither the process nor the methods of peer review are perfect. But what is arguably perfect is the intent of peer review. It exists specifically to try and ensure that, before being accepted, a paper (or book, conference proposal, grant application, etc.) has been thoroughly reviewed — typically by at least two peers who are experts in the field. Peer reviewers are explicitly responsible for evaluating the merit of the work they’ve been assigned, according to the reviewing organization’s criteria. Per a modest adaptation of the Council of Science Editors to include all scholarly disciplines, this includes:
- Providing written, unbiased feedback in a timely manner on the scholarly merits and the value of the work, together with the documented basis for the reviewer’s assertions
- Indicating whether the writing is clear, concise, and relevant and rating the work’s composition, accuracy, originality, and significance for the journal’s readers
- Maintaining the confidentiality of the review process: not sharing, discussing with third parties, or disclosing information from the reviewed work
- Providing a thoughtful, fair, constructive, and informative critique of the submitted work, which may include supplementary material provided to the journal by the author
- Determining the merit, originality, and scope of the work; indicating ways to improve it; and recommending acceptance or rejection using whatever rating scale the editor or other selection group deems most useful
- Noting any ethical concerns, such as any violation of accepted norms of ethical treatment of animal or human subjects or substantial similarity between the reviewed manuscript and any work concurrently submitted elsewhere, which may be known to the reviewer
Peer review is vulnerable to cultural and other biases, as is any human endeavor, and we take that seriously. But its goals of subjecting research to scrutiny, to identifying possible errors or concerns, and to providing this feedback to the authors in such a way that they can use it to materially improve their work, reflects core values. This process may be undertaken privately, as in the case of blinded peer review, or, to a greater or lesser degree, openly, for example, through publication of the review report itself. In some cases, especially in scientific disciplines, authors may also be required to provide access to the underlying data and/or proof of replicability.
There is certainly some evidence to suggest that a more transparent process is both wanted and needed by researchers.
Irrespective of the form of review, providing transparent (i.e., clear, accurate, and easy-to-find) information on their peer review process should be a no-brainer for any journal or organization. This could be as simple as a complete description of how they do peer review to, for example, banning confidential remarks to the editor in favor of a collaborative and consensual discussion between reviewers that ensures the author understands the rationale behind their decision (see, for example, eLife). There is certainly some evidence to suggest that a more transparent process is both wanted and needed by researchers. For example, 82% of respondents to PRE’s 2016 survey want to know if an article has been screened for plagiarism, 81% want to know the number of reviewers, and 74% want information about the peer review method. The same survey showed very strong support for more transparency around publisher policies: 97% agreed (84% strongly agreed) that peer review policies should be freely available; 93% agreed (71% strongly) that journals should provide a description of their peer review process; and 95% agreed (83% strongly) that publication ethics policies should be freely available.
While common sense steps can provide the transparency in peer review and clarity in other scholarly contexts such as research process, including sources and data, it is as important to be transparent about the values inherent in peer review. As an iterative process, peer review relies on a mutual dedication to knowledge on the part of reviewer and author, and on the communal intellectual labor, including that of editors, that makes knowledge creation possible. In other words, peer review is emblematic of broader values and collaborative practices associated with scholarship specifically but also in the production and circulation of knowledge more generally. Good professional journalism, for example, relies on a set of principles that emphasizes the use of sound evidence and fact-checking. What peer review and associated like practices threaten is opinion-based assertions that purport to be facts, as well as ideological inflexibility.
As we’ve argued here, peer review is big. It touches us all — not just as authors or editors or publishers, but as readers and consumers of peer-reviewed scholarship in its immediate and derivative forms. When the March for Science featured signs proclaiming the value of peer review, it might have seemed lighthearted to some. But “What do we what? Science! How do we want it? Peer Reviewed!” is in fact an assertion of core values embodied in a practice essential to our ongoing striving for knowledge.
Other posts this week on the Kitchen for Peer Review Week will consider the challenges of reviewing digital scholarship; self-citation and the past, present, and future of peer review and we will end the week with a report from the Peer Review Congress (taking place September 10-12). We hope you’ll read and engage with these important issues throughout the week.