I say “finally” since my attempts to get the authors to respond to my questions seemed futile. No one was willing to defend the paper, not even the journal’s editorial office.
And yet, the authors did finally respond through a Letter to the Editor, published in The Annals of Thoracic Surgery on July 20. I only heard about this letter from a colleague, not from the authors themselves.
The letter “We Stand By Our Data–A Call for Professional Scholarly Discourse“, by Jessica GY Luc and two co-authors reads with a tone of indignation, taking more offense that I questioned their integrity than their research. So offended, Luc appears, that she would not even cite my blog posts critiquing her research, or even dare write my name. How this letter got past an editor without these necessary details is surprising, until you realize that Luc and her two co-authors, Thomas Varghese and Mara Antonoff, are also editors of the journal. The letter was received on July 14th and accepted just two days later, according to its publication history.
While every surgeon has the right to vent their spleen, the authors make two claims that may require emergency surgery. To avoid any misunderstanding that may come from my rephrasing, I will quote their words verbatim:
Claim 1: We were unwilling to share our raw data — Our raw datasets are our intellectual property, ongoing analyses and longitudinal follow-up are underway, and we are not required to share them with individuals unknown to us who demand them.
Scientific publication is about making scientific claims public. (It’s strange that I’d have to write such an obvious statement to a readership of professionals dedicated to scholarly communication). If Luc wanted to keep her data and results private, she should not have published them in a scientific journal where the scientific community has the right to read, question, and demand evidence underlying her claims.
However, Luc’s dataset was built from public data sources, which was how I was able to reconstruct it (dataset download .xlsx) from a list of papers she provided in an appendix published in an older paper. If Luc was able to detect a very large citation effect after just one year, why was I unable to reproduce their findings? Well, she had an answer for that too:
Claim 7: No citation differences exist between the intervention and control arms of the study upon reanalysis of a reconstructed dataset. This comment is based on a false premise that the outcomes of interest at 1 year are identical to the outcomes at 2.4 years. It is simply impossible for the dataset to be recreated because our analyses were conducted at different timepoints–and Altmetric scores as well as citations are dynamic and changing continuously.
The authors double-down on their defense by arguing that “it is infeasible to retrospectively analyze the citations that were present at the exact timepoints that we performed our analyses.”
Citations are cumulative — they don’t simply vanish from the public record. An average difference of 2.4 citations in favor of the Twitter intervention group in year 1 should should have been detected in year 2. Even so, I downloaded new citation data from Clarivate and plotted them by year of publication. As you can see from the box plots below, there is no evidence that tweeted papers outperformed the control group at any time after the experiment was initiated: Not in year 1, not in year 2, not in year 3 or in year 4.
In spite of counterfactual evidence, if you take time to digest their last claim for a minute, Luc and others are making an argument that it is not sufficient to approximate their study closely; it has to be the exact study. And because they won’t tell you exactly when they measured paper performance (and you won’t be able to do it retrospectively, anyway), you have no grounds to question their results. Plus, they won’t send you their dataset, citing intellectual property rights. Essentially, Luc is arguing that the reason why no-one can replicate their results is that their experiment is, by design, non-reproducible.
Will anyone die or suffer because of this paper? Absolutely not. But the same could be said for the vast majority of scientific papers published each year. The claims made by Luc and others may help justify the jobs of social media professionals in publishing, but they may also whittle away the precious hours of people with the false belief that tweeting benefits the dissemination of research and boosts the citation performance of journals.
The process of trying to communicate with the authors of this paper exposes serious questions about the integrity of the editorial and review process performed at The Annals of Thoracic Surgery. If readers cannot fully trust the claims made about a paper on tweeting, should they trust a paper on open-heart surgery?
The real damage of this paper is not the promotion of a sham intervention, but to the reputation of the journal and the editors who are entrusted to run it. Yes, this is a time for professional discourse. The authors either need to admit error and retract their paper or provide evidence to support their claims.