Recommender systems that promote related articles across publisher platforms increase citations according to a new study.

The paper, “The Citation Advantage of Promoted Articles in a Cross-Publisher Distribution Platform” was published in JASIST online on 23 Dec 2019 by Paul Kudlow, Co-founder of TrendMD.

TrendMD is an automated system that operates directly from journal websites to recommend relevant content across the TrendMD network. Recommended links are based on the content of the paper, the reader’s history, and the history of other readers. It is not unlike the recommendations one would see when online shopping, streaming movies or music. Recommendations appear at the bottom of an online paper following the authors’ references.

Before I start my review, I need to note that four of the five authors of this paper are founders or employees of TrendMD; the fifth was the lead author’s faculty advisor when he was a graduate student. These conflicts of interests do not invalidate the study, only that this paper deserves a closer and more critical evaluation given these relationships.

In other posts, I’ve critiqued research reporting that posting one’s papers to Academia.edu boosts citation performance, that AI software was better at selecting manuscripts than editors, or that Google has resulted in authors citing older papers. I also took issue with a paper written by one of Kudlow’s co-authors, for his study on article tweets and citations. Anyone worried that I won’t take a fair, but critical, approach to this paper should stop reading this post and focus instead on the company’s marketing literature. I’m an analyst, not a promoter.

Remarkable methods

The TrendMD paper is remarkable for several reasons: First, the authors set up their study as a randomized controlled trial with foresight into what they wanted to measure (citations and Mendeley saves), for how long (6 and 12 months), and the necessary size of their dataset to find a difference (sample size calculations). This was not a case of indiscriminately digging up data and mining for significant results.

Like other quality medical research, they followed the CONSORT guidelines for reporting their results. Because of this structure, their results come with greater strength of evidence and a better indication that TrendMD is a cause of, and not merely correlated with, better article performance. This paper is valuable for its methods alone and should be read by other companies interested in researching their efficacy of their products and services.

Unfortunately, the way Kudlow and others analyze their data overstates the benefits of TrendMD and, in some ways, obscures their findings.

Unequal comparisons?

A properly-conducted randomization will result in groups that are similar, in all respects, with each other. This is important because differences in the baseline characteristics of each group could result in performance differences by the end of the study. Without baseline similarity, you can’t be sure whether those differences were the result of the treatment or the groups themselves.

There is some evidence that the control and intervention groups were not equal at the beginning of the study. Papers randomized to the TrendMD arm were published in higher impact journals than the control and had accrued more citations by the start of the study (shown in Table 1 of the paper). This difference was large for some subject categories. For example, TrendMD papers in the Health and Medical Sciences started with an average of 3.1 cites per paper compared to just 1.9 for the control. Kudlow did conduct baseline analysis, he described to me by email, only it didn’t make it into the paper.

Good reporting or strategic marketing?

In this study, outcomes were reported as mean (average) performance, which I found odd, given that most distributions in science communication are highly skewed. Like household income, it makes little sense to report averages, as a very small number of super-rich families greatly distort the rest of the population. This is why the US Census Bureau is adamant about reporting median income performance in its reports. If the distribution of citations were more like, say, the distribution of systolic blood pressure, I would be much more comfortable with accepting average citation performance as a comparison metric. By choosing mean performance to report their results, Kudlow and others may have greatly exaggerated the effect of TrendMD. If the authors were not employees or financially tied to the company, I would consider this oversight to be the result of inadequate statistical training and not strategic marketing.

Overstated results

To his benefit, Kudlow has been open and transparent with his analysis and provided me with additional details. Unfortunately, these details underscore how selective reporting leads to radically different performance measures.

For example, as reported in their paper, mean citation differences between TrendMD and control papers at 12 months were 10.1 vs. 15.2 — a difference of 5.1 citations, on average, or a 50% benefit. Yet, median citation performances were 5 and 6 — a difference of just 1 citation or a 20% benefit. You don’t need to guess which metric made it into the paper’s Abstract.

The authors of the TrendMD study also included a regression analysis, which should have provided the reader with a more appropriate estimate of treatment effect (see Multivariate regression model, Table 6). However, it is not clear from their paper (or with my correspondences with Kudlow) that the authors understood how to properly construct their regression model, report, or interpret their results. TrendMD clearly has a positive effect on citations; we just don’t know how much.

Lacking context: what this paper adds to what we know

There is a long history to the study of how scientific results are distributed and promoted to other researchers and to the lay public. There has also been much work on the information seeking behaviors of readers. Less is known about why researchers cite one paper over another. Clearly, this is a very complex system that involves information science, sociology, economics, communication, government policy, technology, and infrastructure.

While I’m convinced that TrendMD has some beneficial effect on the dissemination of related research (as measured by Mendeley saves and Scopus citations), I have a hard time putting this paper in sufficient context to understand these benefits. For instance, we don’t know how TrendMD compares to other article recommender systems. Does TrendMD’s recommender algorithm do a better job (click-through rate) than other algorithms? Are TrendMD links more likely to be selected than the author’s own reference links? Does placement matter? Does the relevancy of recommendations change as the TrendMD network expands, and is their algorithm biased to provide preferential treatment to some journals, publishers, or authors than others? Without attempting to answer any of these questions, we are left with the efficacy results of a single commercial service taken at a single point in time. Should these papers be considered sound science or just another form of marketing?

Conclusion

The TrendMD paper provides some evidence that they are getting readers to related papers although its effect may have been greatly overstated. It is a methodologically strong paper, but weak on statistical reporting. And while published in a peer reviewed journal, readers should understand that these vendor-promoted studies may contain reporting bias and lack appropriate context.