Yesterday’s release of Altmetric’s 2017 Top 100 Articles list continues the year-end ritual, its coverage of numerous sources concatenated to rank scholarly articles by online attention scores. Given the excitement the release of this list causes every year, it feels akin to a wrapped gift for journal editors, staff, and readers, many of whom open it seeking validation or dreading disappointment.
But what does the list actually tell us? What does it represent?
Altmetric is a scoring system, with weighted scores based on the type of online source mentioning an article. To tally up an Altmetric score, you just add the weighted scores based on event monitoring. Altmetric has added a distinctive design by arranging the sources into a flower with the synthetic metric at the middle.
However, how these weightings are determined is puzzling and opaque. For example, an article appearing in a news outlet receives a weighted score of 8 in the Altmetric system. Why 8? Why not 3 or 10 or 15?
Two things are worth checking with metrics — validity and completeness.
First, is the approach validated? That is, is a Twitter posting (score of 1) really worth 4x a Facebook post (score of 0.25)? Is a Google+ post (score of 1) worth the same as a Twitter post and 4x more than a Facebook post? Why are blogs in general given a score of 5, yet Sina Weibo, a Chinese blogging platform, only given a score of 1? Why is Wikipedia worth 37.5% of News, 12x more than Facebook, and 3x more than Twitter?
There doesn’t seem to be any validation supporting these weighting assignments, and there’s no clear indication that they are validated on a rolling basis. For example, as Facebook has gained prominence and Google+ has faded, the Altmetric weightings don’t appear to have shifted to follow.
Then we get into the question of completeness. There have been numerous changes in media outlets, especially social media outlets, in the past few years. Where are WeChat, Instagram, Snapchat, Qzone, and WhatsApp? It turns out that data from social media platforms, especially those in China, aren’t available to Altmetric, according to Euan Adie, founder of Altmetric. Beyond social media, there are other sources that surveil the literature for professionals in the field and might be worthy of including in Altmetric measures — for instance, publications like Journal Watch and Retraction Watch.
Altmetric finishes gathering data for the Top 100 list in mid-November. This schedule means articles from the prior year carry over as candidates for the next year. In the 2017 list, there are 5 articles from December 2016.
The average age of articles on the Top 100 list is 197 days from publication, with the oldest being 352 days old, the newest a mere 7 days old. The 7-day-old article ranks #50 for 2017 despite accumulating data for only a week before time was up — as we’ll see, most of the effect documented by Altmetric might occur early in most cases, meaning the Altmetric flower may denote a splash rather than a long tail.
Downloading the dataset for the 2017 Top 100 articles, there is a slightly different parsing of the data than the table here would suggest:
- A category titled, “Number of Mendeley Readers” is now included in the raw data behind the Top 100, but according to Adie, these numbers are not factored into the scoring as they are not auditable. They are in the dataset “for extra information.”
- Publons and PubPeer data are apparently captured in a column titled, “Number of peer reviews” (answer? 10 across all 100 articles, with one article accounting for half that number alone).
- F1000 posts are included separately (they hardly contribute at all to the outcomes).
- The number of LinkedIn posts is uniformly zero, which Adie said is due to the fact that LinkedIn data are not available to them. ResearchGate data are another source Altmetric would like to include, but cannot.
Calculate out the weightings and raw data, and you find that 96% of the results are attributable to news stories (54%) and Twitter (42%), with blogs contributing 3%, and the everything else adding up to less than 1%.
Given that Twitter is imprinted heavily with news links, it seems Altmetric is essentially measuring news coverage from its huge list of news outlets, with little activity coming from the other sources. Of the 17 total sources, 13 of them account for just 0.8% of the total for the Top 100.
Remove the weightings, and Twitter accounts for 82% of the total effect, with news accounting for just 13%, and Facebook for 3%. Everything else accounts for 1% or less.
As a raw data measure, Altmetric is mostly measuring Twitter.
Altmetric’s calculations exist in the context of “complex socio-technical systems”
Given the high prevalence of Twitter in either case, it’s worth reflecting that Altmetric’s calculations exist in the context of other algorithms, or what Clifford Lynch mentioned in his recent paper about the stewardship of algorithms as “complex socio-technical systems.” Twitter is a combination of social and technical interactions, with recommendations surfaced based on followers, followers of followers, likes, replies, time of day, and other factors. What you see is not a direct list but a list curated by algorithms that, as Lynch writes:
. . . cannot stand alone: they operate in a very complex and extensive (and often proprietary, unrecordable or even un-reproducible and unknowable) context.
Here are a few socio-technical systems in play with the news and social media, especially Twitter in this case, that may have a bearing on the Altmetric Top 100:
- Journal brands, which are related to prestige, awareness, social media connections, and media profile, as well as number of followers, tweets and retweets, likes, and more
- Media outreach capabilities, which include cultivation of outlets, press releases, coordination with author institutions and their media outreach, and so forth
- Social media marketing techniques, which include dedicated staff, automated placement and measurement tools, expertise, and experience
- Author prominence, which is often related to high profile publication events, since more experienced authors often have larger labs, more resources overall, and greater experience getting grant funding
- Institutional prominence, which includes the size of the organization, its alumni, its location (major metropolitan area with strong local news coverage, or not), and more
All of these factors and more contribute to the social context in which the news and Twitter operate. Getting media coverage for the Lancet or JAMA or NEJM or Nature or Science is much easier than it is for the Journal of Psoriasis. Given the outsized effect of news and Twitter on the Altmetric Top 100 list, journals with savvy media operations, big brands, good social media practices, and prominent authors from large institutions in major metropolitan areas all factor into the scores.
Altmetric is curating its list more carefully this year, eliminating opinion pieces from the list, another change that led to some consternation expressed via Twitter (of course). For example, President Obama’s opinion piece published in Science in January has a higher Altmetric score than anything on the Top 100 list, and would have led to another year of Obama leading the list. It wasn’t the only article to suffer this fate with the editorial change.
In short, given the data behind the Altmetric Top 100 for 2017, it appears Altmetric is basically a measure of news coverage and news coverage amplification via Twitter. The absence of LinkedIn data (all zeroes) and the lack of impact from Facebook does make me wonder about the quality of the data queries of these popular sources for information sharing and news mentions. There is a data availability problem plaguing Altmetric scoring — we don’t know what we don’t know.
Overall, the Top 100 list remains interesting, and perhaps data availability and other elements will improve over time. But the strength of the variables that lead to news coverage and amplification suggests to me that the Top 100 list could well remain a predictable mix of high profile articles from major journals from major researchers from major institutions in major cities with one major platform a major factor. Those seem to be the socio-technical elements driving this particular set of rankings.