How well does TrendMD work to drive eyeballs and citations to your journal? It depends on how you analyze the data.
A recent paper authored by the founders and employees of TrendMD, a recommender system for academic content, reported an overall citation benefit of 50% — a net gain of 5.1 citations within 12 months. I took issue with how the company reported its findings in a way that is consistent with good marketing, just not good science.
After several conversations with the first author, Paul Kudlow, I was able to secure a subset of their dataset for validation purposes. This was the best I was able to achieve as the journal in which their paper was published does not have an explicit data sharing or archiving policy.
The subset represents eight journals publishing in the Health and Medical Sciences. This is an important detail, as it is here where the authors reported the largest citation effect from their study: A mean increase of 20.45 citations per paper, which translates to an 82% benefit. Even a novice to citation analysis will realize that this is a very large effect.
The effect of a single outlier
As it happens, there was one extreme outlier in the TrendMD arm that exerted a huge influence on their overall findings. The paper, published in Circulation, received 1,013 citations over the 12-month observation period (Fig. 1 below). While the dataset did not include article titles, it is pretty clear what this paper was: Heart Disease and Stroke Statistics Statistical Update 2018, an annual report issued by the American Heart Association, which typically receives thousands of citations over its lifetime. Remove this one paper and the red bar in the above figure drops by 5 citation points or 20%.
What is the real TrendMD effect?
First, I need to clarify that by using the word “real” I am not accusing the researchers of data fabrication. By “real” I mean reporting a statistic that is both appropriate to their data and meaningful to current and prospective clients of TrendMD services.
Based on a simple linear regression that normalizes the distribution of citations (i.e. converts it from a highly skewed distribution to a bell-shaped distribution and therefore minimizes the influence of outliers), we arrive at an overall citation benefit of 17% (Table 1), still positive and significant, but a far cry from the reported effect of 82%. Seven of the eight journals demonstrated a positive effect, while one estimate was negative. Don’t read too much into the exact performance of each journal as we expect high variation given the limited size of each journal subset.
Rich get richer
In spite of the high variation in results, there are some general trends in the data that may help us understand how TrendMD works and whether preferential bias is built into their algorithm.
Papers published in higher impact (JIF) journals appear to receive a stronger citation effect. Based on a more elaborate regression model, the TrendMD effect increases by 0.5% for each 1 point increase in JIF. Given that there are many cumulative advantage (Matthew) effects known to operate in science, how does this affect TrendMD’s recommendation system?
Opening TrendMD’s black box
Even after controlling for the number of click-throughs (readers clicking on a recommended article link from the TrendMD widget), the number of impressions (the number of times an article is recommended to a reader) remains a significant predictor of citations. How can this be?
From their paper, Kudlow and others describe that the impression predictor could be explained through indirect effects: Just by seeing a recommended paper, a user may be more likely to read it, save it, and cite it at some unspecified later date. To me, this seems highly speculative when there is a more direct explanation.
Preferential bias built into the algorithm
While we don’t know exactly how TrendMD’s recommender system works — it is essentially a black box — we know from the company that it includes three components: 1) keyword overlap between papers, 2) the reader’s clickstream history, and 3) the clickstream history of other readers.
However, neither #1 (keyword matching) nor #2 (reader’s clickstream) can explain the residual effects of link impressions or why higher impact journals received a stronger TrendMD effect. But #3 (community clickstream) can.
For analogy, consider a restaurant recommendation app on your phone that operates similarly to TrendMD. You search for “Thai food” and the recommendation algorithm finds kitchens in your vicinity that include words from the restaurant name (e.g. Taste of Thai, Bangkok Express) and menu choices (Pad Thai, Tom Yum, Pla Goong). The recommendation also knows that you search for Thai food regularly (who wouldn’t?). However, the recommendation system also weights your results based on what other users have searched and rated. When it comes to restaurants, users really want to know where other people dine and enjoy, which is why this component is featured so prominently on apps like Yelp. Recommendation drives traffic and traffic influences recommendation.
So, if TrendMD is recommending papers based on community clicks, it is recommending papers published in more popular, frequently-read, wide-distribution journals. Put another way, TrendMD is biased toward recommending papers from prestigious journals. To explain their results, Kudlow and others don’t need to speculate on what readers are doing when not using TrendMD; they simply need to look more closely at how their algorithm is working.
After reanalyzing a subset of TrendMD’s dataset, I am still confident in the efficacy of this product: It appears to work, just more moderately than reported by the company. Nevertheless, most media companies would be thrilled with an effectiveness of 17%, as studies of the effect of social media on readership and citations report few (if any) effects.
Placed in context, TrendMD may have developed a highly effective tool for driving traffic to relevant content. What makes it different from other products with recommendation systems (PubMed, Scopus, Web of Science, ProQuest, EBSCO, among others) is that TrendMD operates from within the content itself, obviating the need for keeping readers engaged in a large siloed platform. This platform-neutral model allows TrendMD to grow organically and, in theory, benefit from scaled network effects. TrendMD also allows customers to actively promote their content in the recommendations, similar to paid search ads showing up within Google’s organic results or sponsored product recommendations on Amazon. Publishers should take interest, only cautiously and skeptically.
This is my own recommendation.