Each June, editors, publishers, and authors anxiously await the release of the Journal Citation Report (JCR)–a dataset that reports, among other things, Journal Impact Factors for approximately 12,000 scholarly publications.
While new titles are added to the JCR each year, several dozens will be suppressed for “anomalous citation patterns.” Stated in more direct language, the JCR will kick out journals that attempt to game the system. Thomson Reuters, the publishers of the JCR, prefers to use language that does not imply intent; however, the result is the same. Last year, 38 titles were delisted from the JCR: 23 for extremely high rates of self-citation and 15 for “citation stacking” a term that may also be construed as a citation cartel.
As a tool, title suppression has strong moderating effect on citation behavior. Journals reinstated in the JCR after suppression have self-citation rates that are in-line with other journals in their field, and citation cartels (you cite my journal and I’ll cite yours) are stopped abruptly. The damage to the reputation of a journal is apparent for journals that slink back into the report. Even the threat of being suppressed is powerful enough to prevent editors from tempting fate. My clients tell me this all the time. The potential damage to their title (and their own reputations) is just too great to justify the risk.
This year, Scopus, a large, cross-disciplinary literature index product owned by Elsevier, announced that it would begin a similar annual evaluation process, directed to “maintain content quality” in its dataset. Scopus data is used to generate annual citation metrics in its SCImago Journal & Country Rank (SJR) product–a competitor to the JCR.
Like the JCR, Scopus will consider delisting a title if it engages in high levels of self-citation and provides a benchmark figure: 200% beyond self-citation rates of peer journals. In this way, Scopus provides some transparency in how it will make decisions but avoids establishing arbitrary cross-the-board levels. Scopus will let your discipline decide what is appropriate citation behavior. In contrast, the JCR is focused on how self-citation changes a journal’s ranking within a field. When a journal’s rank is radically shifted by self-citation, the JCR may use this as grounds for suppression.
Editors often ask me how much self-citation is too much, and I respond (honestly) that I don’t know. “Can you give me a ballpark?” (no). “Do you think this will be detected?” (I can’t say). What are self-citation rates in comparable journals (I can provide you with that!). While these answers are unsatisfactory to many editors, there is power in ambiguity. No editor wants to be the one responsible for having his journal suppressed.
When Scopus was first released in 2004, its main feature was scope, hence the name. At that time, the market was saturated with disciplinary-based indexes. The dominant interdisciplinary index at that time, the Science Citation Index (now part of the Web of Science) was more concerned with quality and selectivity than with comprehensiveness. If you wanted to be part of the Web of Science, you should expect a long and rigorous evaluation process. Not so with Scopus.
In the following years, the Web of Science responded by growing the size of its dataset and therefore the number of journals reported in its annual JCR metrics report. Last year, the JCR included 11,022 unique journal titles, about 50% more than in 2004. Amid this expansion, it did not relax its editorial standards for inclusion. Curating the Web of Science–and the journals included in its JCR–is as important today as it was 20 years ago.
By announcing their intention to police the content indexed in Scopus, and the data it generates for its own annual journal metrics report (SJR), Elsevier has signaled a similar intention toward content curation. At a time where more comprehensive indexes are provided to researchers for free (e.g. Google Scholar), comprehensiveness just doesn’t cut it, especially if you want to derive valid, authoritative metrics from your data.
Google Scholar may counter that it is possible, through their algorithms, to detect “anomalous citation patterns” but it takes humans to investigate, make decisions, take responsibility, and remain ultimately accountable for their decisions. Computers cannot do that: you need people, people with expertise, and people whose expertise is respected. Not surprising, this costs a lot of money, which is why Google Scholar takes an algorithmic approach to data curation.
Personally, I find Elsevier’s SJR a far superior product to Thomson Reuter’s JCR; however, none of my clients have ever asked me to use SJR data. The SJR reports many of the same metrics as the JCR, for example, Cites per Doc (2yr.) measures the average citation performance of papers over a two-year window (example record here), which is exactly the same as the Journal Impact Factor. The SJR also reports self-citation rates among other metrics of interest to editors and publishers. Did I mention that the SJR is free?
What accounts for the continued dominance of the JCR in the citation metrics market, especially when faced with excellent free alternatives? In addition to data quality, historical persistence may play a role.
Established indicators (and the companies that produce them) persist, in part, because they provide longitudinal context. Many economists and financial professionals loathe the Dow Jones Industrial Average, a simple leading indicator first calculated by hand in 1896, and yet there is a huge value in continuing with the same indicator because it provides an historical perspective on how the economy is performing over time. Likewise, journal editors and publishers want to know how their journal is performing compared to last year and whether they are trending upward or downward. While other citation indicators may prove to be more valid and reliable, the Journal Impact Factor (and its name) provide that anchoring.
It appears that Elsevier understands this issue and is also in the process of indexing and adding pre-1996 material into Scopus. With older citation data, Scopus will also be able to calculate author h-indexes and other indicators that require having an author’s entire publication history.
In sum, the announcement that Scopus has established quality indicators suggests that they are now willing to invest in the most costly resource in building authority–people. In a world of abundant and cheap data, there is a real and growing demand for authority.
Correction: The original version of this post incorrectly stated that Impact Factor was a US trademarked term held by Thomson Reuters.