Calculating the number of journal papers that reach the top-cited 1% is more informative than the Impact Factor (IF), argue the editors of Nature Biomedical Engineering in an editorial, “Challenge the impact factor,” which was published on 11 July 2017.
Their alternative to the Impact Factor is a new index they call the “Impact Quotient” or IQ for short. Straightforward in its construction, the IQ is the percentage of a journal’s papers that reached the top 1% most-cited papers in the journal’s research area. There is a lot to like about a citation metric that is simple, intuitive, and can be explained in a single unqualified sentence.
The editors’ figure, a scatterplot of IQ against Impact Factor reveals a correlation between the two measures for high performance journals. This is not altogether surprising as journals with high Impact Factor scores tend to publish a good number of highly cited papers.
But what about the bottom left corner of the plot where the vast majority of the journals in their dataset are found?
I investigated this for the field of microbiology for reasons that will make the methodology for calculating IQs much more complicated than reported in the editorial. The Essential Science Indicators (ESI) — the data source from which the editors extracted their list of highly-cited papers — is very different than the Journal Citation Report (JCR) — the source that reports Impact Factors. While both are produced by Clarivate, the former lists just 22 generic subject categories while the latter reports 236 specific subject categories. For example, there are 14 distinct engineering categories in the JCR but only 1 in the ESI.
Even for Microbiology, in which both data sources list just one category, the journals are not the same. The ESI includes multidisciplinary science titles (Nature, Science, PNAS, Nature Communication, Scientific Reports, and PLOS ONE); the JCR does not. Remove these titles and just 72 (59%) of the 123 microbiology titles included in the JCR would get an IQ score above 0%. Put another way, 41% (51) of the microbiology journals that received a 2016 Impact Factor would receive an IQ of exactly 0%.
Unsurprisingly, most of the highly cited papers were published in prestigious, high Impact Factor journals.
More importantly, of those 72 microbiology journals that would receive an IQ score above zero, 14 published just one highly cited paper and 8 published two. Unsurprisingly, most of the highly cited papers were published in prestigious, high Impact Factor journals, like Nature Reviews Microbiology (IF 26.819) and Cell Host & Microbe (IF 14.946).
For journals with Impact Factors smaller than 10, the editors claim that there is no correlation between IQ and IF, which, in theory has the potential of adding new information to the performance evaluation of a journal. However, the IQs of these journals are based on a very small number of papers that made it into the top-1% of their field. As a result, for all but a small group of elite journals, the one-percent club is a small numbers club, prone to chance and subject to wild fluctuations from year to year.
For all but a small group of elite journals, the one-percent club is a small numbers club, prone to chance and subject to wild fluctuations from year to year.
While I like the IQ in theory, in practice, it is unable to differentiate a large swath of research journals, is unable to distinguish performance among specialist fields, and for journals with IF>10, provides us with the same information as the Impact Factor. To me, the IQ is hardly an alternative to the Impact Factor. Nevertheless, it may serve as another performance data point that could be added to the JCR‘s panel of metrics.
It’s not hard to understand why Nature Biomedical Engineering editors like the IQ: Nature journals fare quite well in the index. Indeed, the editors highlight Nature journals in their plot.
In the end, we are left with an index, far more limited in scope, and unable to add any reliable information to what we already know: If you wish to read high impact papers, you are likely to find them in high impact journals.