Last week, Slate published an ill-advised hatchet job by an education columnist specializing in the humanities. The topic? Peer review. While the particulars of the Slate article inspired me to add a sharply critical comment, and I was only one among dozens who found fault with the article, there is a fundamental question worth considering in all this:
What is “peer review”?
Peer review certainly isn’t one thing. Arguing as if it were is a fundamental error made by the Slate columnist and often by others. Peer review is constantly evolving. It is actually difficult to define precisely because of all the variations it can have. All this makes general approbation or condemnation of peer-review difficult to take without the requisite grain of salt.
We often make an “availability error,” generalizing to all forms of peer review based on the limited information we have about the kinds we’ve experienced. Combine this with how peer review seems captive to history — as if it were invented once and then taken forward like a precious gemstone — and it’s easy to see why there’s confusion about how variable peer review can be, how it’s evolved, and where it’s going.
Here are some variables around peer-review we have to understand before we know what kind of peer review we’re actually talking about:
- Is it blinded?
- If it is blinded, is it single-blinded or double-blinded?
- Is there statistical or methodological review in addition to external peer-review?
- Are the peer reviewers truly experts in the field or a more general assemblage of individuals?
- What are the promises and goals of the peer review process?
- What type of disclosure of financial or other potential competing interests is made? Are reviewers aware of these?
- Is there a senior editor of some sort involved along with outside peer reviewers?
- Is the peer-review “inherited” from another body, such as a committee or a preceding journal process (e.g., in “cascading” title situations or when expert panels have been involved)?
- Are there two tiers of peer review within the same journal’s practices?
- Is the peer-review done at the article level or at the corpus level (as happens with some supplements)?
- Is plagiarism-detection software used as part of the process?
- Are figures checked for manipulation?
- Is the peer reviewer graded by a senior editor as part of an internal evaluation and improvement process?
Each one of these points is worthy of discussion, and the myriad combinations of these variables constitute peer review in the large sense. However, using any particular combination as a single encapsulation or representation of peer review is akin to using a single automobile model to stand in for all cars ever and to come. You may have a Pinto on your hands, you may have a Tesla, or you may have a Camry. Each one shares some common features, but what matters are the differences.
What is the ideal peer review process? It depends on the goals and intent of the journal and its editors. This is another place where generalizations can get in the way. Medical journals generally have very high standards for peer-review, while journals that don’t deal with health decisions or life sciences can have different standards and requirements without performing any disservice to readers. Computing journals, humanities journals, and others develop their own standards and should take pride in what they do if their readers value and trust the output. There is no single formula. There is, as the pediatricians say, “a wide range of normal.”
So-called “post-publication peer review” is an interesting animal in its own right. As I’ve written about before, there are many problems with equating commenting with peer-review, including a lack of pre-qualification and no central evaluative authority or accountability. And it’s nothing new, in a sense. The most basic form of post-publication peer-review is called “reading.” A more involved version is called “a discussion” or “a journal club.” The more elaborate version yet is called “science.” Those who want to capture it in computer code are making essential trade-offs, many of which we don’t understand completely yet. One of these is the cost of entry into public post-publication peer-review systems — the effort it entails, the risk for publicly being shamed for saying something ill-advised or clearly wrong, and the lack of actual give-and-take. Again, post-publication peer-review is not one thing, and just because you capture some aspect of it in commenting software doesn’t mean you’ve done anything much.
Peer review is a category of activities, not a singular activity. In track and field, there are the running events, but this is a category of events. The sprint events differ dramatically from the distance events, which are blended in the team relays. Even within the sprint events, for example, there are important differences between the specific types (outright sprint vs. hurdles), just as there are middle-distance events and long-distance events.
Peer review, as a category, is also changing, and quite rapidly. In the past decade, we’ve added image manipulation tests, plagiarism detection software, new disclosure rules, and a growing roster of reviewers from around the world thanks to online submission systems and email. Now, publishers can consider signing on with transparency services and preliminary review services. Peer-review is evolving, sometimes rapidly, sometimes slowly, but progress is being made.
With this degree of variability and the constant evolution it’s undergoing, peer review is difficult to praise or criticize uniformly. It is more appropriate to critique certain aspects of it.
However, in critiquing it, we also encounter our own expectations of what it can provide. Some people believe peer review should find all possible errors or misinterpretations in or around a study. Others believe peer review can stop marginal studies from being published. What peer review does is limited, important, and useful. There is nothing wrong or incompatible with those aspects. Even the most rigorous peer review at a journal cannot stop a study from being published somewhere. Peer reviewers can’t stop an author from self-promoting a published work later. Peer reviewers do not check all the datasets, rerun calculations of p-values, and so forth, except in the cases where statistical reviewers are involved — and even in these cases, statistical reviewers often check the methodologies used, sample some data, and move on.
Failures we attribute to peer review often belong to authors. If someone can name a so-called “peer review scandal” that doesn’t boil down to authors making outlandish claims, fabricating data, or misinterpreting their own data, please point it out. Blaming a process may help us avoid confrontation or accusation, but it also can mislead us to fix something that requires nothing more than sensible updating and improvements while allowing authors to abdicate responsibility.
Does peer review work? Is peer review broken? The vast majority of authors believe it improves their final work, and since it’s evolving from this solid base, it’s clearly not broken. But before we can have a useful discussion about its purpose and effectiveness, we need to agree on which approach to peer review we’re talking about, then whether our expectations of it are reasonable and accurate.