There are other important value metrics beyond the strength of a journal. This might come as a shock to some STEM publishers, who have flourished or floundered based on the performance of impact factor rankings published each June. While the value of the journal as a container is an important value metric and one that needs to continue, the rapidly evolving alternative metrics (altmetrics) movement is concerned with more than replacing traditional journal assessment metrics.
Like much of life these days, a key focus of our community has been on those qualities that are measured to ensure one “passes the test.” The coin of the realm in scholarly communications has been citations, in particular journal citations, and that is the primary metric against which scholarly output has been measured. Another “coin” for scholarly monographs has been the imprimatur of the publisher of one’s book. Impact factor, which is based on citations, and overall publisher reputation provide the reading community signals of quality and vetting. They also provide a benchmark against which journals have been compared and, by extension, the content that comprise those titles. There are other metrics, but for the past 40+ years, these indicators have driven much of the discussions and many of the advancement decisions during that time. Given the overwhelming amount of potential content, journal and publisher brands provide useful quick filters surrounding quality, focus, interests, or bias. There is no doubt that these trust metrics will continue. But they are increasingly being questioned, especially in a world of expanding content forms, blogs, and social media.
At the Charleston Conference last week, I had the opportunity to talk with fellow Scholarly Kitchen Chef, Joe Esposito about altmetrics and its potential role in our community. Joe made the point, with which I agree, that part of the motivation of some who are driving new forms of measurement is an interest in displacing the traditional metrics, i.e., the impact factor. Some elements of our industry are trying to break the monopoly that the impact factor has held on metrics in our community so that newer publications might more easily flourish. Perhaps the choice of the term “altmetrics” to describe the movement implies the question, “Alternative to what?” — which leads one back to the impact factor and its shortcomings. The impact factor is an imperfect measure, a point even Eugene Garfield acknowledges. We needn’t discuss its flaws here; that is well-worn territory.
The inherent problem with that focus is that it misses a key point of assessment about the actual impact of a particular piece of research (or ultimately its contributor) that is represented by one or more individual articles that may have been published in multiple journals. Our current metrics in scholarly publishing have been averages or proxies of impact across a collection (the journal), not the item itself, or the impact across the work of a particular scholar or particular research project. The container might be highly regarded, and the bar of entry might have been surpassed, but that doesn’t mean that any particular paper in a prestigious journal is significantly more valuable within its own context than another paper published in a less prestigious (i.e., lower impact factor) title. The fact that there are a growing number of papers that get rescinded is a signal of this even within the most highly regarded titles.
Assessment is increasingly important to the communities directly related to, but not part of, the publishing community. Yes, libraries have been applying usage-based assessment and impact factor for acquisitions decisions for some time, but there is more they could/should do to contribute to scholarly assessment. But beyond that, the academe and the administration of research institutions rely on publication metrics for researcher assessment, promotion and tenure decisions. Grant funding organizations have used the publication system as a proxy for assessing grant applicants. However, these are only proxy measures of a researchers impact, not directly tied to the output of the individual researcher.
Scholarly communications is also expanding in its breadth of accepted communications forms. Scholars are producing content that gets published in repositories and archives, blogs, and social media — separately from or in addition to journals. Some researchers are publishing and contributing their data to repositories such as ChemBank and GenBank. Others, such as in the creative arts, are capturing performances or music in digital video and audio files that can be shared just like journal articles. Traditional citation measures are not well-suited to assessing the impact of these non-traditional content forms. If we want to have a full view of a scholar’s impact, we need to find a way to measure the usage and impact of these newer forms of content distribution. Addressing these issues is the broader goal of the altmetrics community and it is perhaps clouded by the focus on replacing the impact factor.
In order to make alternative metrics a reality, there must be a culture and an infrastructure for sharing the requisite data. Just as there was opposition by segments in the publishing community to the creation of standards for gathering and sharing of online usage data in the late 1990s, there exists opposition to providing data at the more granular level needed for article-level metrics. It is instructive to reflect on the experience of our community with COUNTER and journal usage data. When Project COUNTER began, usage data reporting was all over the map. In addition to wide inconsistencies in how data was reported between publishers, there were many technical issues — such as repeated re-loading of files by users, double-counting the download of a PDF from an HTML page, and downloads by indexing crawlers — that skewed the statistics, making them unreliable.
Fortunately, tools are quickly falling into place to provide metrics on the things that the larger scholarly community really needs: individual items and individual scholars. The broader scholarly community has been aware of these needs for some time, and we are making considerable progress on implementation. There has been a great deal of progress in the past decade in increasing the granularity of assessment metrics, such as the h-index and the Eigenfactor. These metrics are gaining traction because they focus on the impact of a particular researcher or on particular articles, but they are still limited by dependence solely on citations. Newer usage-based article-level metrics are being explored such as the UsageFactor, Publisher and Institutional Repository Usage Statistics (PIRUS2), both led by COUNTER, as well as other applications of usage data such as the PageRank and the Y-Factor. Additionally, infrastructure elements are becoming available that will aid in new methods of assessment, like an individual researcher ID through the ORCID system that officially launched in October. Other projects described in Judy Luther’s posting on altmetrics earlier this year are examples of infrastructure elements necessary to push forward the development of new measures.
Our community needs to move these pilots and discussions of alternative metrics into the stage of common definitions and standards. We need to come to consensus on what should be included and excluded from calculations. For example, we need clear definitions of what constitutes a “use” of a dataset or software code and how to quantify the applications of data from within a larger collection. Some of these determinations might be domain specific, but many of these issues can be generalized to broader communities. Because these resources can and often do reside in multiple repositories or resources, thought needs to be given to how metrics can be consistently reported so that they can be universally aggregated. Additionally, we will need commitment to be a level of openness about the automated sharing of usage data so that network-wide measures can be calculated. Efficient methods of real-time (or nearly real) data collection should also be considered an important element of the infrastructure that we will come to expect. While a central repository of data for anonymization purposes or for more robust analytics might be valuable, it probably isn’t a necessity, if we can reach agreement on data supply streams and open data-sharing tools and policy guidance.
By all means, it is early days in developing alternative metrics and the technical and cultural support structures needed. Some have argued that it might be too early to even begin establishing best practices. However, if alternative assessment is to really gain traction, agreeing at the outset on the needed components of those metrics will solve many downstream issues and lead to more rapid adoption by everyone who needs better assessment of the value of scholarly communications.