Impact factors are both loved and despised.
Some argue they reflect little more than the entrenched hegemony of the established science journals conveying their prestige on the select few articles allowed to pass through their hallowed editorial gates. Others argue that articles are cited on their scientific merit alone and that citations have nothing to do with the source in which they are published.
The problem with understanding how much prestige a journal conveys on its articles is that journals are in the business of publishing unique articles. Comparing the citation performance of articles published in one source against the next always comes down to comparing apples and oranges.
One way of getting around the apples to oranges problem is by seeking out cases of unethical multiple publication — identical (or nearly identical) articles published in multiple sources. Another way is to track the performance of articles meant to be published in several sources.
In an article published this month in the Journal of Clinical Epidemiology, Thomas Perneger, a Swiss epidemiologist, tackles the question of impact factors and citations by analyzing the performance of medical consensus statements — public reports published simultaneously in multiple journals. Perneger focuses on just four statements published 33 times:
- QUOROM (QUality Of Reporting Of Meta-analyses) – published 3 times
- CONSORT (CONsolidated Standards Of Reporting Trials) revised – 8 times
- STARD (STAndards of Reporting of Diagnostic accuracy) – 14 times
- STROBE (STrenghening of the Reporting of OBservational studies in Epidemiology) – 8 times
By comparing the citation performance of these statements compared to the journal’s impact factor, Perneger hoped to investigate the degree to which a journal’s prestige is conveyed on its articles. By using exact copies of articles, he was able to control for article characteristics (e.g. topic, authors, length) as well as variation in article quality, the bugaboo of most impact factor studies.
Not only were citation performances of the consensus statements highly correlated with their journal’s impact factor, Perneger reports, their relationships were linear — each logarithm unit increase in impact factor predicted one logarithm unit increase in citations, meaning that if one journal’s impact factor was twice that of another, you could predict a doubling of article citations. Perneger discusses the meaning of these results:
. . . the analysis presented here suggests that the association between journal impact factors and future citations is in part self-fulfilling prophecy, as publication in a high-impact journal will push upward the citations of any given article, independent of its value. This creates an additional but perverse incentive to pursue publication in high-impact journals.
Perneger’s study, while admittedly small in size, adequately addresses article to article differences. His set-up is simple and elegant; his is results are straightforward and intuitive.
Not so fast.
Perneger demonstrates a correlation between article citations and the prestige (as measured by impact factor) of the journal. If prestige were driving an author’s decision to cite one version of a consensus statement over another, the author would need to know that each version of the article existed in the first place, and then make a conscious decision to cite based on the prestige of the source. For instance, an author required to cite the STARD statement would need to know that it was published in 14 different sources, have an intuitive sense of the pecking order of the journals and select accordingly.
Does this sound reasonable?
The main weakness of this paper is the inability to separate the citing process from the information discovery process. Highly cited journals tend to have higher circulation and broader readership in the scientific community. If a consensus statement was published in a multidisciplinary medical journal like JAMA, The Lancet, or BMJ, it’s likely to get much more readership than if it were published in a smaller, specialized journal. Likewise, someone who does a general web search for the statement is likely to land on the larger and more prestigious of the journals.
In other words, it is just as plausible that Perneger’s results are explained by enhanced visibility than by prestige alone.
Until we get into the minds of the authors while they are preparing their manuscripts, we may never really know why one article gets cited over another . . . although it’s still fun trying.
Perneger, Thomas V. 2010. Citation analysis of identical consensus statements revealed journal-related bias. Journal of Clinical Epidemiology 63: 660-664, DOI: 10.1016/j.jclinepi.2009.09.012