Editors’ note: Today’s post is by Sami Samiei Esfahany, a geodesist and senior researcher at the UK Centre for Observation and Modelling of Earthquakes, Volcanoes and Tectonics (COMET) and the University of Leeds. Reviewer credit to Chef Lisa Janicke Hinchliffe.

Prelude

About ten years ago, I defended my PhD at Delft University of Technology in the Netherlands. As part of the Dutch PhD tradition, candidates submit, alongside their dissertation, ten propositions: short, opposable statements—intended to invite challenge — that they should be prepared to defend. One of mine was:

“Novelty and innovativeness should be given greater priority than clarity and comprehensiveness in peer review.”

Appropriately enough, one member of the defense committee challenged this proposition during my defense. I do not intend to revisit here whether the proposition, as I formulated it then, was right or wrong. But the question behind it stayed with me. Over the decade since that defense, through my experiences as an author, reviewer, and researcher, it has gradually evolved into a broader concern: do we judge familiar and unfamiliar science by the same standards?

That is the question I want to explore here.

The Problem Behind the Proposition

Scientific peer review is one of the foundations of modern science. Its primary purpose is to determine whether a manuscript meets the scientific standards and scope required for publication. Although no review system is perfect, the scientific community generally accepts that occasional mistakes are inevitable. Strong papers are sometimes rejected, while papers that arguably fall below a journal’s intended standards are sometimes accepted. But what if these two kinds of mistakes are not simply occasional, but are systematically encouraged by the way different types of manuscripts are reviewed?

Peer Review as a Statistical Decision Problem

A useful way to think about this problem is through an analogy with statistical hypothesis testing. In statistics, two types of decision errors are well known:

• Type I error (False Alarm): rejecting a true hypothesis.
• Type II error (Missed Detection): failing to reject a false hypothesis.

A similar framework can be applied to scientific publishing. For simplicity, suppose submitted manuscripts can be broadly divided according to whether they satisfy the publication standards of a particular journal:

• Publishable papers, whose scientific quality and contribution justify publication according to the journal’s scope and standards.
• Non-publishable papers, which do not sufficiently meet those standards.

This distinction is necessarily journal-dependent: a paper appropriate for a specialist journal may fall below the significance threshold of a highly selective multidisciplinary journal. Every journal therefore applies, explicitly or implicitly, some form of publication threshold. Manuscripts judged to meet that standard are accepted; those judged not to meet it are rejected. This naturally leads to two analogous errors:

• False Acceptance (FA): accepting a manuscript that does not sufficiently meet the journal’s intended publication standards.
• False Rejection (FR): rejecting a manuscript that does meet those standards.

The Expected Trade-Off Between False Acceptance and False Rejection

If the review process applies a single, consistent standard to every submission, these two errors should exhibit a familiar trade-off. Making the journal more selective should decrease false acceptance but inevitably increases the risk of false rejection. Making it less selective should have the opposite effect.

Yet many researchers feel that this trade-off does not fully explain what they observe. Instead, it can sometimes appear that both false acceptance and false rejection are occurring at substantial rates.

How can this happen?

Different Manuscripts, Different Effective Thresholds

The simple model assumes that the same publication standard is effectively applied to every manuscript. In practice, however, the threshold may depend partly on how familiar the topic is to reviewers.

Highly innovative papers often introduce unfamiliar concepts, methods, or interpretations and are therefore inherently more difficult to evaluate. Greater uncertainty may lead reviewers to demand additional evidence or validation, effectively raising the threshold for acceptance. Conventional or incremental papers, by contrast, are generally easier to assess because their methods and conceptual frameworks are familiar. This does not make incremental research less valuable or less deserving of publication, but familiarity may make reviewers more comfortable recommending acceptance and less likely to question whether a borderline contribution meets the journal’s intended standard.

The concern, therefore, is not that incremental papers are accepted while innovative papers are rejected, though that may be the effect. The concern is that familiarity may make reviewers more likely to accept papers that would otherwise fall below the journal’s intended standard, while unfamiliarity may make them more likely to reject innovative papers that actually meet or exceed that standard.

A Systematic Asymmetry in Peer Review

This familiarity effect creates the possibility of a systematic asymmetry: innovative papers that genuinely satisfy the journal’s standards may face a higher probability of false rejection, while familiar or conventional manuscripts that are borderline with respect to those standards may face a higher probability of false acceptance. If this hypothesis is correct, peer review is not operating with a truly uniform decision threshold; the effective standard depends partly on the nature and familiarity of the contribution.

This problem may not affect all journals equally. Highly selective multidisciplinary journals such as Nature or Science explicitly place strong emphasis on novelty, broad significance, and potential impact, making marginal but familiar contributions less likely to progress through the editorial process. Of course, these journals also differ substantially in their editorial structures, screening procedures, and review processes, so any difference cannot be attributed to their emphasis on novelty alone. Indeed, these differences may themselves help protect against some of the asymmetries discussed here.

Mainstream technical journals with narrower disciplinary scopes may be particularly susceptible to this asymmetry. Where technical correctness and consistency with established literature carry substantial weight, familiar contributions may be easier to assess and validate, potentially increasing the risk of false acceptance for borderline papers. Conversely, unconventional contributions may be more difficult to evaluate within familiar frameworks, potentially increasing the risk of false rejection. The same review environment could therefore contribute to both errors simultaneously. This has important consequences for scientific progress. Science advances through a combination of careful incremental work and occasional conceptual breakthroughs. Both are necessary.

The concern is not that the scientific literature contains incremental research, but that systematically rewarding familiarity while penalizing unfamiliarity could make some contributions easier to publish while creating additional barriers for others. In some cases, if such barriers persist across multiple journals, they could even distort what ultimately enters the scientific record.

Why Better Reviewers Alone Are Not the Solution

The natural response might be to demand “better reviewers.” However, this is probably not the complete solution. The real issue is not reviewer quality alone, but the assumption that every manuscript should be reviewed in exactly the same way.

We would never apply the same statistical test to every dataset regardless of its characteristics. Likewise, physicians do not prescribe the same treatment to every patient simply because they share a diagnosis. Different situations require different decision strategies. There is little reason to expect that scientific peer review should be any different.

Instead of a uniform review process, peer review should become adaptive.

The Need for Adaptive Peer Review

The first and perhaps most important responsibility therefore falls on editors to recognize the nature of the contribution before selecting reviewers. Reviewer selection is particularly important.  A technical verification, an incremental methodological improvement, a new theoretical framework, or an unconventional interdisciplinary study may require distinct types of expertise and review strategies. The editor’s role therefore extends beyond simply coordinating referee reports.

Incremental studies can often be effectively assessed by specialists with focused technical expertise, including early-career researchers. In contrast, genuinely novel or unconventional contributions may benefit from reviewers with broader scientific vision, greater experience, and openness to ideas that do not fit neatly within established frameworks.

This does not mean that innovative papers should face an easier review. They should be assessed just as rigorously, but with a form of scrutiny that can distinguish genuine innovation from unsupported speculation without penalizing unfamiliarity itself.

Strengthening Editorial Leadership

Ultimately, the objective of peer review is not simply to reject weak manuscripts but to identify work that deserves to enter the scientific record. This requires applying high standards consistently while recognizing that different contributions may require different expertise to evaluate them fairly.

Equally important to achieving this is fostering a stronger culture of editorial leadership. Peer review should not become a purely mechanical process in which the recommendations of two or three reviewers automatically determine the final decision. Reviewer reports provide essential technical expertise, but they should inform, not replace, the editor’s own scientific judgment. This is particularly important for unconventional, interdisciplinary, or potentially transformative work, where reviewer opinions may be more diverse and uncertainty inherently greater. In such cases, editors should be prepared to consider the broader scientific context, seek additional perspectives when necessary, and make independent decisions that balance technical rigor with scientific originality.

The goal should therefore not be to make peer review more permissive toward innovative work or more restrictive toward incremental work, but to judge both by their scientific merit rather than by their familiarity. Peer reviewers will always make mistakes, but a system that systematically finds familiar work easier to accept and unfamiliar work easier to reject risks selecting for conformity rather than quality. Preventing this requires active editorial judgment, thoughtful reviewer selection, and a willingness to give unconventional ideas the level of consideration they deserve.

Looking back at the proposition I defended ten years ago, I would probably formulate it differently today. The issue is not that novelty and the capacity for innovation should necessarily be given greater priority than clarity and comprehensiveness. Rather, novelty should not make a paper more vulnerable to rejection. And the review process should deliberately guard against placing an additional burden on novel contributions. Science progresses not only by confirming what we already know, but by recognizing what we have not yet learned to expect. Peer review should be designed to protect that possibility.

Author’s note (AI disclosure): OpenAI’s ChatGPT was used to improve clarity and for language editing.