An Old-Age Problem Among Reviewers?

According to Nature News, a study was presented at the Sixth International Congress on Peer Review and Biomedical Publication in Vancouver, Canada, that sought to show that peer-reviewer quality deteriorates with age. Unfortunately, if the Nature News article is a fair representation of the research (which is unpublished), the study seems to fall short of demonstrating much of anything.

The study was conducted by Michael Callaham, MD, editor-in-chief of the Annals of Emergency Medicine. He compiled and analyzed the scores editors at the journal had given to more than 1,400 reviewers between 1994 and 2008. Annals editors rate reviews on a scale of one to five, with one being unsatisfactory and five being exceptional. Ratings are based on whether the review contains constructive, professional comments on study design, writing and interpretation of results, providing useful context for the editor in deciding whether to accept the paper.

While average scores stayed the same at 3.6 on the scale of 5, more than 90% of reviewer scores fell during the period at an overall rate of 0.04 points per year. This leads Callaham to believe that reviewers get worse as they grow older:

Callaham agrees that a select few senior advisers are always very useful. But from his own observation, older reviewers do tend to cut corners. He notes that psychological research shows that experts in complex tasks typically reach a plateau and then stay there or slowly deteriorate. Perhaps by the time researchers are asked to review a paper at his journal, they are already experts. He suspects the same would hold true for journals across all fields.

It’s important to remember there are at least two moving parts to this scoring system — the reviews submitted and the editors reviewing the reviews. Therefore, there are several possible explanations for Callaham’s modest observations:

1. Younger reviewers are less familiar with the conceits of peer-review, so they write long reviews full of minuscule details and tangents, unnecessary but commendable embroidery as they learn the ropes.
2. Younger reviewers are trying to impress, so spend more time and care on their reviews.
3. Younger reviewers are less familiar to the editors, so they appear as “fresh faces” and are scored higher as a reflection of novelty and to encourage them to review again.
4. Older reviewers have been seen again and again by the editors, so are less likely to impress on the second or third time reviewing, especially assuming they are being seen by the same subspecialty editor.
5. Older reviewers are reviewing more often, for more publications, and know the ropes. They do “cut corners” or use more jargon or bottom-line language to get their views across.
6. During the timeframe under consideration, the evaluation tool became more familiar to users. The tool also “aged” as the users and reviewers did, a factor that can’t be ignored. Teachers at the beginning of their careers grade differently than they do later, and it’s likely editors who gained experience with the tool graded differently as time passed
7. The editors at the Annals of Emergency Medicine probably aged the same amount as the reviewers over the years. Are the editors at Annals becoming grumpier graders?

Overall, attempts to make scientific peer-review more scientific seem misguided. The purpose of tools like these eludes me. If it’s to create a more stable, replicable system, that seems to occur when good people with sound judgment agree about what they’re doing and create a great journal. Quantitative measures of their intramural behaviors are pretty irrelevant, and even distracting.

But maybe I’ve been doing this too long, and I’m just getting older . . .