Impact factors are both loved and despised.
Some argue they reflect little more than the entrenched hegemony of the established science journals conveying their prestige on the select few articles allowed to pass through their hallowed editorial gates. Others argue that articles are cited on their scientific merit alone and that citations have nothing to do with the source in which they are published.
The problem with understanding how much prestige a journal conveys on its articles is that journals are in the business of publishing unique articles. Comparing the citation performance of articles published in one source against the next always comes down to comparing apples and oranges.
One way of getting around the apples to oranges problem is by seeking out cases of unethical multiple publication — identical (or nearly identical) articles published in multiple sources. Another way is to track the performance of articles meant to be published in several sources.
In an article published this month in the Journal of Clinical Epidemiology, Thomas Perneger, a Swiss epidemiologist, tackles the question of impact factors and citations by analyzing the performance of medical consensus statements — public reports published simultaneously in multiple journals. Perneger focuses on just four statements published 33 times:
- QUOROM (QUality Of Reporting Of Meta-analyses) – published 3 times
- CONSORT (CONsolidated Standards Of Reporting Trials) revised – 8 times
- STARD (STAndards of Reporting of Diagnostic accuracy) – 14 times
- STROBE (STrenghening of the Reporting of OBservational studies in Epidemiology) – 8 times
By comparing the citation performance of these statements compared to the journal’s impact factor, Perneger hoped to investigate the degree to which a journal’s prestige is conveyed on its articles. By using exact copies of articles, he was able to control for article characteristics (e.g. topic, authors, length) as well as variation in article quality, the bugaboo of most impact factor studies.
Not only were citation performances of the consensus statements highly correlated with their journal’s impact factor, Perneger reports, their relationships were linear — each logarithm unit increase in impact factor predicted one logarithm unit increase in citations, meaning that if one journal’s impact factor was twice that of another, you could predict a doubling of article citations. Perneger discusses the meaning of these results:
. . . the analysis presented here suggests that the association between journal impact factors and future citations is in part self-fulfilling prophecy, as publication in a high-impact journal will push upward the citations of any given article, independent of its value. This creates an additional but perverse incentive to pursue publication in high-impact journals.
Perneger’s study, while admittedly small in size, adequately addresses article to article differences. His set-up is simple and elegant; his is results are straightforward and intuitive.
Case closed?
Not so fast.
Perneger demonstrates a correlation between article citations and the prestige (as measured by impact factor) of the journal. If prestige were driving an author’s decision to cite one version of a consensus statement over another, the author would need to know that each version of the article existed in the first place, and then make a conscious decision to cite based on the prestige of the source. For instance, an author required to cite the STARD statement would need to know that it was published in 14 different sources, have an intuitive sense of the pecking order of the journals and select accordingly.
Does this sound reasonable?
The main weakness of this paper is the inability to separate the citing process from the information discovery process. Highly cited journals tend to have higher circulation and broader readership in the scientific community. If a consensus statement was published in a multidisciplinary medical journal like JAMA, The Lancet, or BMJ, it’s likely to get much more readership than if it were published in a smaller, specialized journal. Likewise, someone who does a general web search for the statement is likely to land on the larger and more prestigious of the journals.
In other words, it is just as plausible that Perneger’s results are explained by enhanced visibility than by prestige alone.
Until we get into the minds of the authors while they are preparing their manuscripts, we may never really know why one article gets cited over another . . . although it’s still fun trying.
—
Perneger, Thomas V. 2010. Citation analysis of identical consensus statements revealed journal-related bias. Journal of Clinical Epidemiology 63: 660-664, DOI: 10.1016/j.jclinepi.2009.09.012
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=9e4c31f5-cefc-481a-8de9-449a06304202)
Discussion
8 Thoughts on "Impact Factors — A Self-fulfilling Prophecy?"
Trying to explain statistics is indeed the fun part. You make a good point that discovery probably plays a big role in citation, a role that is not well studied. The scientometrics community tends to be statisticians not psychologists.
Citations are often gathered after the fact, so leading journals are a good place to look, in fact many articles feature multiple cites to just a few top journals.
But it is not clear that this positive feedback is perverse or constitutes bias. One uses citations to demonstrate the importance one’s own work, so citing top journals is both easier and promotional.
One of my favorite studies around this was published in 1991. The authors studied NEJM papers published before, during, and after the infamous 1978 New York newspaper strike, a 3-month long affair that removed a lot of high-end medical reporting from the media of the time. They found that papers covered by the New York Times received 70%+ more citations, and followed it for 10 years.
The media of today is completely different, of course, and blogs, Facebook, Twitter, and sharing matter as much or more. Also, search engines have probably mitigated the power of media coverage in creating awareness. I’d love to see a study in a decade or so covering social media, SEO, and citations.
Good point. The media can certainly amplify the role of the journal in disseminating research.
Several years later, Kiernan generalized the effect of media amplification on article citations to include other newspapers, although there was no effect from networked TV.
Kiernan, V. 2003. Diffusion of news about research. Science Communication 25: 3-13, http://dx.doi.org/10.1177/1075547003255297
This is a fascinating chicken-and-egg sort of conundrum. Do prestigous journals get more visibility because of their higher IF or does an IF increase as a result of high visibility?
I am inclined to believe the former, but I agree that an article’s citation frequency in a particular journal is probably decided by discoverability/visibility…which, presumably, is brought about by the journal’s prestige. So the author may not necessarily compare IFs before choosing a journal to cite, but the IF is indeed what ultimately causes the author to cite a paper in one particular journal over another.
If it is a feedback process then each causes the other, so the often asked question as to which causes which is simply misplaced. Also, it is not the IF per se that acts here, but what the IF measures, namely citations.
If a journal is highly cited then authors of important (for now) papers will want to publish there, and it is these important papers that will be highly cited. (What sometimes gets lost in the discussion is that there really are important papers.)
As described, the journal system tends to stabilize over time, an equilibrium model. It is my understanding that IF’s change slowly, although Phil would know more about that. What makes the system change is that science changes over time. Research is a lot like mining, in that new idea seams are found, worked and abandoned, the famous S-curve of interest in a topic. Journals compete for market share by betting where science is going.
My team has experimented with using a disease model to try to get at some of these dynamics. It shows that speed of communication, including publicity, is indeed a critical factor in the growth of new fields. (Interestingly, no one seems to want to study the death of fields, which is just as important from a journal’s point of view.)
See: http://www.osti.gov/innovation/research/diffusion/#casestudies
Although this is not specifically discussed in this post, could part of the complexity be eliminated if journals stopped publishing consensus statements and other such documents in multiple journals? I’ve never been a fan of deliberate publication in multiple places, especially in this era of Google, PubMed, bookmarking sites, and sharing via social media.
Such intentional duplicate publication seems to be anchored in the days of print-only publishing, when the discoverability of a study was limited to those people having access to a print copy of the specific journal something appeared in — in that instance, publishing in multiple journals ensured that more people would see it. Is this true still today? Perhaps, but at what cost?
Haven’t the sophisticated search engines, databases of literature, and social bookmarking and media tools obviated the need to publish the same thing in multiple journals? It’s frustrating to perform a search and find a document listed in 5 different journals, and for authors it must be frustrating to then decide which one to read and which one to cite if they end up citing it.
Is publishing it once, in one place, enough?
Whatever the mechanism is, the hypothesis the article set out to explore was proved, surely? i.e. that publishing an article in a higher impact journal leads to higher citations. And I am not sure it makes sense to try and distinguish between “prestige” and “visibility” anyway – they are intertwined aspects of a journal.
Excellent point.
There are some publishers (e.g. PLoS) advancing the notion that these concepts “prestige” and “visibility” can be disassociated, and by tracking the metrics of an article, one can arrive with the true value of that article.
The point of my post (and your comment) was that even in near-ideal experimental conditions, it is difficult to distinguish how value is conferred from the journal to the articles published within.
