Tweeting by a network of journal social media editors increases citations and Altmetric scores, reports a new paper published in the Annals of Thoracic Surgery. Not surprisingly, it is also getting a lot of tweets.

The study is based on a randomized controlled trial, in which 112 original research papers were assigned to either the intervention arm (receiving tweets and retweets by members of the journals’ social media network delegates) or no intervention at all. The authors measured each paper’s citations and Altmetric scores after one year.

Yet, a closer reading of this paper yields substantive questions and concerns about its veracity, including the source of citations, measured outcomes, and an inability to validate or recreate the dataset. Neither the authors, nor the editorial office, have been willing to explain or defend this paper.

While the authors report a sizable difference in citation performance after just one year (an average of 3.1 additional citations to the treatment papers vs. just 0.7 citations to the control papers), the authors don’t state their source of citations. This is curious, as the results may vary widely depending on whether they used data from the Web of Science, Scopus, CrossRef, Dimensions, PubMed, or Google Scholar.

The authors also count Altmetric scores for each paper in their trial, but there are two glaring problems here: First, Altmetric scores are composites of many social media events that include tweets. So, if the intervention was tweeting, Altmetric scores of tweeted papers will increase ipso facto. Second, both journals use PlumX Metrics not Altmetric, so it’s not clear how they got article-level Altmetric scores.

There are other statistical oddities in this paper that made me curious about its veracity, such reporting the Odds Ratio (OR) on continuous variables like citation frequency.

I contacted Mara Antonoff, corresponding author of this paper and Assistant Professor at the MD Anderson Cancer Center on June 10, with the above questions and was promptly sent a response that she, or the first author, Jessica Luc, Cardiac Surgery Resident at the University of British Columbia, would get back to me shortly with answers. One week later, with no response, I sent a reminder. Another week later, I contacted the editorial office of the journal and asked for their involvement plus a copy of the authors’ dataset. The office responded with an invitation to submit a formal letter to the editor, which may, or may not, be accepted for publication. Both Luc and Antonoff are listed on the editorial board of the journal and all authors of this paper are listed on the journal’s list of Thoracic Surgery Social Medial Network (TSSMN) Delegates. Yet, there is no author disclosure section to this paper.

Given a lack of response from the authors and limited involvement from the journal’s editorial office, I attempted to recreate the authors’ dataset. According to their methods, 56 articles were randomized into the treatment (tweeted) arm of the study. Given that tweets are public, I looked into the tweet history for Antonoff, Luc, and other members of the TSSMN social media delegates but could only identify 23 tweeted articles made during the period described in their paper. I asked the authors about this discrepancy and Antonoff responded by email, “As we stated in the manuscript, the study included 112 articles, among which 56 were tweeted.” She ignored my second request for their dataset. The Annals has no stated data availability policy. I asked the editorial office a second time for their involvement and was told to submit a letter-to-the-editor.

Authors of scientific papers should be able to explain and defend their research and provide any additional evidence to support their claims. In turn, the editor of the journal has authority over published papers and can take action if authors are unwilling to comply. Such actions may include an Editorial Expression of Concern and, if necessary, a retraction. The Annals of Thoracic Medicine is a member of the Committee on Publication Ethics (COPE), which does provide direction on how to respond to concerns sent directly to to the editor. Reminding the journal editor of their COPE membership did not change his position how to communicate my concerns.

In sum, this paper makes bold claims about the effect of collaborative tweeting on the performance of medical papers. Not surprisingly, this paper has been widely tweeted, shared on social media, and is used as strong evidence, given that it is claimed to be based on a randomized controlled trial design. Prior studies show little (if any) effect of social media effects on paper performance.

Unfortunately, the paper contains serious omissions and neither the authors, nor the journal’s editorial office, are willing to explain or defend this paper. More alarmingly, there is a lack of evidence that the study was performed as described. Unlike a medical trial, in which some data may be kept confidential to preserve the anonymity of patients, I don’t see a similar argument for a study that utilized published papers and public Twitter accounts. Moreover, the hesitancy for the authors to respond, given that several are also members of the journal editorial board, is alarming; the lack of concern from the editorial office is even more concerning. Without public access to the data behind this paper, the results of this study should be considered suspect.

[An update to this investigation was published on 13 July 2020, Reanalysis of Tweeting Study Yields No Citation Benefit.]