There are some things that get better with age and experience. Reviewing manuscripts, unfortunately, is not one of them.
A long-term study of reviewer quality in a medical journal reports a small but significant decline in the performance of peer-reviewers over time.
Their paper, “Longitudinal Trends in the Performance of Scientific Peer Reviewers,” by Michael Callaham and Charles McCulloch, appeared earlier this year in the Annals of Emergency Medicine.
Callaham and McCulloch analyzed the quality scores of nearly 15,000 manuscript reviews performed by 1,500 reviewers between 1994 and 2008. The quality of each review was rated on a five-point scale by one of the journal’s editors. Callaham and McCulloch were primarily interested in the rate at which individual reviewers’ quality scores changed over time.
As a group, reviewer quality scores declined steadily by about 0.04 points (or 0.8%) per year, a small but significant change. Not all reviewers faired so poorly, however: 8% of reviewers improved their scores over time, but 92% of them got worse. Even the performance of the best reviewers showed general declines over time (a decrease of 0.03 points per year).
Overall scores of the reviewers stayed constant over time despite individual deterioration as newly recruited reviewers came in with higher scores than those who had been removed from the reviewer pool.
Callaham and McCulloch propose two explanations for the observed decline in reviewer performance:
- Decreased cognitive abilities of reviewers as they age, which include impaired decision-making, unwillingness to comply with all the review requirements, and the inability to keep up to date with current knowledge and techniques.
- A loss of motivation for producing high-quality reviews as reviewers take on additional roles and responsibilities over time. They write:
Competition for the reviewer’s time (which is usually uncompensated) increases with seniority, as they develop (more enticing) opportunities for additional peer review, research, administrative, and leadership responsibilities and rewards.
Personally, I find their loss of motivation hypothesis more convincing than cognitive decline. Or perhaps, as I push into middle age, I wish this to be the case.