From a news perspective, the recent report by Google researchers that more highly-cited papers are found in non-elite journals is about much more than the distribution of citations.
The idea that digital publishing may be helping to raise the awareness of specialized and geographically-isolated journals–along with the authors who publish in them–has a sense of fairness and egalitarianism built into the storyline. The notion of a weakening elite cadre of journals also awakens our political disdain for publishing plutocracies and the scientific bias that may result from it. There is some part of us that wishes science to become more democratic.
When a paper like this comes to light, there is a always a temptation to begin debating its broader context before we can vet its methods. Like the paper claiming that scientists are reading fewer papers, journalists rarely put much effort into validating even the most basic numbers underlying the authors’ bold claims. It is much more newsworthy to discuss the paper’s significance.
Instead of ruminating about the broader economic, sociological and technological changes that may underly Google’s findings, I’m going to focus on on their methods because I believe that their findings may simply be an artifact of the researchers’ strict definition of what constitutes an “elite” journal.
The Google team defined an “elite” journal as one of the top 10 most-cited journals (based on their h-5 index) for each of their 261 subject categories and for each year between 1995 and 2013. Google researchers then identified the 1000 most-cited articles in each category-year and looked to see how many were published in the elite versus non-elite journals.
Google’s method is simple and intuitive, but it contains two fundamental assumptions that they don’t bother to verify in their paper.
First, the use of a top-ten list assumes that the production of scientific literature is constant from year to year. It isn’t. If we assume that the number of articles grows approximately 3% per annum, and we want to conserve the ratio between elite and non-elite journals, then by 2013, there should be 17 journals in their “elite” group not 10.
Second, the Google researchers also assume that elite journals publish roughly the same number of articles each year, which also doesn’t seem to be valid, at least for medicine.
Below is a plot of what most of us would consider “elite” medical journals. With the exception of Nature Medicine, this group of titles shrank by an average of 78% articles between 1995 and 2013. Two journals (JAMA and BMJ) currently publish less than half the number of articles than in 1995. Taken together, these eight elite journals eliminated 1644 possible publication slots.
If we think of this model as a game of musical chairs–where the chairs are publication slots in elite journals and the children circling them are prospective articles–there are more children playing this game each year and fewer chairs to fill when the music stops.
I’ll admit that the analogy of musical chairs to describe the process of slotting articles into journals isn’t entirely correct. In musical chairs, luck has a lot to do with whether you’ll be close to an open chair when the music stops–that, and some shear forcefulness. Publication in an elite journal, however, is not entirely a game of chance. Editors are skilled at spotting the best articles and have a pretty good idea of whether an article will be cited, but they are not perfect. Editors are people and people are subjective. Good articles will slip through their selection process and get published elsewhere. Authors, alternatively, may give up on the elite journals if they know their manuscript will be stuck in the review and revision process for some time. Everyone is trying to make the best decision with never enough information. But when the number of available publication slots in elite journals shrinks each year, the odds are simply stacked against authors.
In sum, an expanding annual production of scientific articles coupled with a limited (or shrinking) number of publication opportunities in the elite journals will invariably lead to more highly-cited papers being found in non-elite journals. If the Google researchers wish to make claims about the distribution of articles published in elite versus non-elite journals, then they needed to account for the annual growth of the literature as well as the changing nature of elite journals.