The Scholarly Kitchen

Proceedings of the National Academy of Sciences

Should members of the National Academy of Sciences be permitted to control the peer-review process for their own submissions?

If one is concerned with citation impact, the answer is both “no” and “yes.”

A recent article, “Systematic Differences in Impact across Publication Tracks at PNAS,” is based on a citation analysis of nearly 2,700 papers published in the Proceedings of the National Academy of Sciences (PNAS) between 2004 and 2005.  The study appeared on December 1st in the online journal, PLoS ONE, and is authored by David Rand and Thomas Pfeiffer, both at Harvard University.

Articles submitted to PNAS currently follow three submission tracks:

1. Direct – authors submit their own manuscripts, which are then subject to double-blind review
2. Communicated – authors submit their manuscripts to an NAS member, who is responsible for overseeing the peer-review process
3. Contributed – NAS members may find their own reviewers and submit their own articles

While direct submissions are the norm for most scientific journals, this method is relatively new for PNAS, adopted as recently as 1995.  Since then, direct submissions have become the norm, with communicated submissions waning in popularity.  In order to simplify and streamline the submission process, PNAS will eliminate communicated submission in July 2010.  There has been some scrutiny of the communicated submission track, some arguing that it has allowed unacceptable papers to be published without adequate review.

Citation statistics seem to support this notion.  As a group, Contributed articles significantly underperformed Direct and Communicated articles.  But when Rand and Pfeiffer looked at the distribution of these three submission tracks, they uncovered something more surprising.

The top 10% of member Contributed papers significantly outperformed both Direct and Communicated papers, while the bottom 10% of member Contributed papers greatly underperformed Direct and Communicated papers.  In other words, there was much more diversity in the citation performance of papers submitted by NAS members.

In explaining their results, Rand and Pfeiffer consider whether the Contributed track (where NAS members select their own reviewers and submit their own work) may subject their papers to a softer and more lenient form of review.  This at least would explain the underperforming articles.

But what about the stellar articles?  Rand and Pfeiffer speculate:

[It is] possible that these alternative publishing procedures may facilitate the publication of time-sensitive and groundbreaking work which is of high quality but might suffer under the standard review process.

In other words, an alternate publication track may counter-balance the overly-conservative review practice of high-prestige journals.  In considering the policy implications for journal publishing, Rand and Pfeiffer maintain that there are real benefits for this model:

The benefit of facilitating publication of extremely high-impact Contributed papers could be argued to out-weigh the potential cost of allowing more low quality papers to also be published.

Confusing peer-review with quality of submission?

Rand and Pfeiffer’s main assumption in comparing the three groups of published papers (Direct, Communicated, and Contributed) is that they are similar in all respects.  Indeed, the researchers attempted to control for effects that are known to be associated with higher rates of citations (paper topic classification, author-pays OA, special feature).  Rand and Pfeiffer admit that they would liked to have included press releases, but they are missing the biggest difference in their analysis — namely, Contributed papers are submitted by a small and elite group of established authors who are invited members of the National Academy of Sciences.

Each year, NAS members may elect a scant 72 new members and 18 foreign associates based on recognition of their outstanding past achievements in research.  Once nominated, these names are subject to a complex and rigorous process of evaluation and election.  Those who become an NAS member, and are therefore able to contribute their own work to PNAS, are very unlike the profile of authors who submit through the Direct track.  Rand and Pfeiffer’s analysis contains no controls for author effects such as previous publication and citation performances, or even the number of authors per paper — all variables that are highly predictive of future citations.

We also don’t know if NAS members are selectively submitting only some of their work to PNAS, or if they are opting to use the Direct submission route, as some members do.

Without controlling for author effects, Rand and Pfeiffer are unable to disentangle the effect of author quality from peer-review process, and thereby confusing one for the other.

While this paper is methodologically sound, the lack of any real effort to consider other possible explanations is the real weakness of this study.  It does, however, pose some important questions about member-facilitated peer-review that should be addressed in more detail.