A new paper on measuring the impact of microblogging via Twitter (tweets) on article citations is making a buzz within the alt-metrics community, with splashy headlines appearing in many of our Twitter feeds:
Cite and retweet this widely: Tweets can predict citations! Highly tweeted articles 11x more likely to be highly cited http://t.co/dcJLRj7y
The article (and the above tweet) were published recently by Gunther Eysenbach in the Journal of Medical Internet Research (JMIR). Eysenbach also serves as editor and shareholder of JMIR.
The article, “Can tweets predict citations? Metrics of social impact based on twitter and correlation with traditional metrics of scientific impact,” has been tweeted 527 times.
In his paper, Eysenbach analyzes a cohort of 55 articles published in JMIR between 2009 and 2010 and tracks their performance — in terms of tweets and citations — to investigate whether the former could predict the latter. Short answer: they could. However, we need to look at this paper in more detail than 140 characters allows in order to understand what is being reported and what it means.
Relying upon data collected by his journal’s Twitter API, Eysenbach focuses on tweets that contain an embedded link to the article, like the URL listed in the tweet above. Since the API registers the date and time when the tweet was published, Eysenbach was able to look at temporal trends in tweets very shortly after the article appeared. Citation data for each article were collected from two sources: Google Scholar and Scopus. At the time of analysis, articles were between 17 and 29 months old.
Not surprisingly, he reports that most tweets were sent out on the first or second day after an article was published followed by a rapid decay. Eysenbach explores the frequency and distribution of tweets over different periods and attempts to correlate them with citation data. Tweets show moderate correlation strength with Google Scholar, he reports, but not with Scopus, a fact that Eysenbach explains by the fact that Google Scholar indexes many non-article sources.
The main message of the paper is that highly tweeted articles were 11 times more likely to be highly cited, a result that makes a great 140 character headline but needs much more context for interpretation.
“Highly tweeted” and “highly cited” articles fall into the top 75th percentile when ranked by frequency of tweets and citations, respectively. There were 12 highly tweeted articles, 9 of which were also highly cited. Conversely, just 3 of the 43 less-tweeted articles were highly cited. While the math for calculating the odds ratio looks pretty formidable, we should recognize that this is a very blunt instrument for measuring article performance. Given the highly-skewed distribution of tweets and citations, the top 25th percentile comprises a very large performance range.
Eysenbach did attempt to construct a continuous regression model including time and tweets as citation predictors and reported that he could explain 27% of total citation variation, a figure that is in the same predictive ballpark as article downloads.
Unlike an article citation — which requires an author to produce a new piece of research, have it vetted by peers, and published in a journal, which is indexed by a reputable source that tallies citations — there are very low barriers to microblogging. Accounts on Twitter are free, and a post requires little more than a few words and a link. Retweeting takes even less effort — a simple click of a button. And if that clicking is too strenuous, many twitter accounts are set up to automatically retweet posts that come their way. It should not be surprising that the 527 tweets to the JMIR article contained many repeat posts, some verging on the compulsive:
- Gunther Eysenbach (3)
- Richard Smith (2)
- Brian S. McGowan PhD (28)
- J Med Internet Res (7)
- HOT Most Tweeted (6)
- RT @ (>100)
What’s more, many of these counted tweets were not sent out by humans. The Journal of Medical Internet Research sends out an automatic tweet when a paper first appears and then sends out monthly tweets to promote the journal’s most tweeted papers. Tweets promoting the journal’s most viewed, most purchased, and most cited articles (from Scopus and Google Scholar) are also sent out automatically, many of which are then retweeted by other tweet bots (and human bots) to the blogosphere. That the author decided to count these tweets as measures of “buzz” leaves me concerned about what Twitter metrics measure and whether they can be considered a valid indicator of article impact.
I have deeper reservations about this paper.
In the shadows of an ongoing legal drama that pits a former editor of a scientific journal against its publisher and the news organizations that attempt to cover it, I’m leery of editors who view their journal as a publication outlet for their own work. While Eysenbach selected his own journal for this study, he decided against outsourcing the editorial and peer-review process. Had this been done, some of these methodological and interpretive problems may have been addressed and potential ethical conflicts could have been avoided. For instance, consider the following paragraph within the methods section:
For the tweetation-citation correlation analysis, I included only tweets that referred to articles published in issue 3/2009 through issue 2/2010—that is, tweetations of all 55 articles published between July 22, 2009 and June 30, 2010 [31-97].
What is wrong with this paragraph? First, Eysenbach cites 66 papers and not just the 55 papers included in his dataset. His reference list thus includes a total of 69 articles citing JMIR, only three of which cite articles for their content — 55 serve to cite data points, and 11 are unaccountable. The above paragraph contains enough information for the reader without serial self-citation. If listing each paper was important for understanding the paper, the author could have listed them in a data appendix. At least, this is what I imagine an external editor and reviewers may have recommended.
Whether or not this practice of citing data points is considered normal in medical research is beside the point: The practice of serial self-citation by an author simultaneously serving as editor and shareholder of his journal appears as suspicious behavior. The effect of this behavior on JMIR’s Impact Factor will become apparent next June when Thomson Reuters issues its 2011 Journal Citation Report. Serial self-citation can result in being delisted from Thomson Reuter’s Journal Citation Report.
Eysenbach has also purchased several domain names (twimpact.org, twimpactfactor.org and twimpactfactor.com) “with the possible goal to create services to calculate and track twimpact and twindex metrics for publications and publishers.” While I appreciate that he discloses his conflicts of interest, with so many sources of potential bias, it becomes hard to separate the paper’s contribution to science from its contribution to entrepreneurship.
Update: 4 Jan, 2012
Gunther Eysenbach has issued a correction to his JMIR paper removing 67 self-citations to his journal. The paper was edited to reflect the change.