Just as sap flowing in my maple tree alerts me of Spring in upstate New York, I can also tell the season by the type of consulting questions that come my way. For me, Spring means predicting journal Impact Factors before they bloom in June.
The work is routine, detailed, but not altogether challenging. If I’m lucky, I succeed at bundling more elaborate calculations, tests and analysis into my consulting. I could survive as a statistical consultant without predicting journal Impact Factors, but I’m not going to give it up while the work still lasts. The work comes because Thomson Reuters (the publishers of both the Web of Science and the Journal Citation Report) is not adequately addressing the needs of publishers and editorial boards; and in that void, consultants like me find a business niche.
In this post, I’m going to describe how the journal Impact Factor is calculated and highlight its major weaknesses, but mostly focus on how Thomson Reuters’ citation services could be reconfigured to more adequately satisfy the needs of publishers and editors.
The journal Impact Factor (IF) measures, for any given year, the citation performance of articles in their second and third year of publication. For example, 2013 Impact Factors (due for release in mid-June, 2014) will be calculated using the following method:
IF2013 = 2013 citations to articles published in 2011 and 2012 / articles published in 2011 and 2012
Simple enough? Let’s get into its complications:
1. What counts as an article? While the numerator of the Impact Factor includes all citations made to the journal in a given year, the denominator of the IF is composed of just “citable items,” which usually means research articles and reviews. News, editorials, letters to the editor, and corrections (among others) are excluded from the count. Yet, for some journals, there exists a grey zone of article types (perspectives, essays, highlights, spotlights, opinions, among others) that could go either way.
The classification of citable items is determined by a number of characteristics of the article-type, such as whether it has a descriptive title, whether there are named authors and addresses, whether there is an abstract, the article length, whether it contains cited references, and the density those cited references (see McVeigh and Mann, 2009). As Marie McVeigh once described this process to me, “If it looks like a duck, swims like a duck, and quacks like a duck, we’re going to classify it as a duck.”
This approach attempts to deal with the strategy of some editors of labeling research articles and reviews as editorial material, yet it leaves room open for interpretation and ambiguity. And where there is ambiguity, some publishers will contest their classification, because fewer citable items means a higher Impact Factor score.
2. Two methods for counting citations. Authors routinely make errors when citing other papers. When sufficient information is provided, a correct link between the citing and cited papers can be made and the citation is counted in the summary record in the Web of Science. When insufficient or inaccurate citation information prevents resolution to the target article, it may still be counted in the Journal Citation Report, which relies on just the journal name (along with common variants) and year of publication. According to Hubbard and McVeigh (2011), the Journal Citation Report relies on an extensive list of article title variants–10 title variations per journal, on average, with some requiring more than 50 title variants. Because of the two distinct methods for counting citations, it is not possible to calculate a journal’s Impact Factor with complete accuracy by using article citation data from the Web of Science.
3. Reporting delays and error corrections. Waiting six or seven months to receive a citation report may not have been onerous when the Journal Citation Report began publication in 1975; however, it does seem increasingly anachronistic in an age where article-level metrics are being reported in real-time. Thomson Reuters’ method of extracting, correcting and reporting metrics as a batch process once per year, impedes the communication of results and propagates citation errors. When discovered, errors found in the Journal Citation Report need to be addressed, corrected and reported in a series of updates. As one editor confided in me after working to have their Impact Factor corrected, “The damage [to our reputation] has already been done.”
4. Reliance on the date of print publication. For most scientific journals, publication is a continuous process; however, many journals list two separate publication dates for their articles: an online publication date and a print publication date. The difference between online and print publication dates can range from several days, for some titles, to over a year in others. For articles that reach their citation peak in year four (rather than in years two or three), delaying the print publication date can substantially inflate a journal’s Impact Factor, research suggests.
Furthermore, as the Impact Factor is based on calendar year of the final publication, there is a temptation to hold articles back from entering their two-year Impact Factor observation window if you think they are likely to gather more citations later in their life cycle. I’ve occasionally heard accusations that editors purposefully post-date their journal issues or push issues from December to January specifically to gain an extra calendar year for their articles.
Summary of problems with the current system
The problems with the current WoS-JCR model can be summed up as:
- Ambiguity over article classification
- Separate citation counting systems
- Delay in reporting performance measures
- Reliance on print publication dates
As a result of these fundamental issues, we are left with a two-product solution that causes general angst and uncertainty, allows guys like me to exploit that uncertainty as a business, and creates an opportunity for some academics to make a career out of ridiculing the system.
I don’t know of any altmetrics advocate who believes that citations are not worth measuring, so I’d like to spend the rest of the time proposing a one-product solution for Thomson Reuters which, in the long run, will make their product much more useful and valuable to the scientific community and to those who wish to measure its publication performance.
Imagine a system where article-level citation data in the Web of Science are used to create real-time journal-level statistics and reports based on the date of online article publication, rather than the calendar year of print publication.
In order to deal with citation error, the onus of providing correct reference information should rest squarely in the hands of the publisher: If you don’t provide us with correct citation data, we can’t count your citations. In this one-product solution, authors would also be incentivized to report errors in the database. Feedback forms are currently provided in the Web of Science, but they are buried and do not encourage wide participation. Incentivizing authors to seek out and correct reference errors to their own papers would eliminate the need to create a separate citation counting system. To actively encourage accurate reference data, imagine, as the corresponding author of a paper, receiving the following email:
Congratulations on having your recent article indexed in the Web of Science. Please take a few minutes to check whether your title, author list, abstract and references are all listed correctly. This link will take you directly to your record and provide you with the tools to suggest any needed corrections. We will also provide you with free 24-hour access to our author metrics section so that you can see how your article is being cited by others, calculate your h-index, and provide you with other useful tools.
An accurate Web of Science citation database would then allow publishers and their editors to view, in real-time, the performance of their journals based on the date of online article publication. While the system would still calculate annual Journal Impact Factors, the date of online publication could also be used to provide users with a continuous (rolling) observation window. Journals that do not issue print or post-date article publication would not be put at a disadvantage in this model.
Acts of gaming would be more transparent and easier to detect. Based on a real-time reporting system, the detailed citation profile of your competitor’s titles become as transparent as your own. A rapid and unexpected rise in a competitor’s rolling Impact Factor, for example, may signal an atrocious case of self-citation, or the emergence of a citation cartel. In the current two-product model, an editor, spotting a rise in a competitor’s 2013 Impact Factor in June 2014 (when the Journal Citation Report is released), might need to comb through an entire year’s citation records in the Web of Science to spot the offending article published nearly a year and a half earlier. In a one-product model, identifying and reporting cases of citation gaming becomes much more immediate.
A one-system solution would also reduce the ambiguity in how Thomson Reuters classifies “citable items” as the calculated metric would be linked back to the articles that composed its construction. Impact Factors based on just the performances of research articles could be calculated on the fly by eliminating the contributions of reviews, editorials, letters, among other citing–but rarely cited–document types (see recommendations 11 and 14 of the DORA declaration). My personal opinion is that Thomson Reuters should abandon the classification system altogether and base classification as a “citable item” on whether the article includes any citations itself.
Last, a single product, based on a database of linked citations could provide users with visualization tools that create journal maps from citation data. With such data, we would be able to see the flow of citations to, and from, each journal in the network, identify important (central) journals from the graph, understand the relationships among disciplines (clusters), detect titles with high levels of self-citation and uncomfortably close citation relationships among two or more journals (cartels).
In conclusion, the Journal Citation Report is based on a process that made perfect sense in 1975 but looks more and more archaic each year. Combining two citation services into a single product would make explicit the way citation metrics are calculated and create a suite of new, dynamic tools directed at the author, editor and publisher. Such a revamping of the two-model system would require some major changes to a company that currently enjoys a strong and loyal customer base; nevertheless, most would agree that if we were to conceive and build a new citation reporting service today, it would look very little like the current system.
Until this product changes, other companies will continue to nibble on the heels of Thomson Reuters, develop new products that address its limitations, or like me, create livelihoods on doing what they could be doing themselves.