A hypothesis without data is valuable if it is based on theory and sound reasoning. We call these conjectures.
Last Monday, I explored a conjecture proposed by Frank Krell, that journals are manipulating their impact factor by post-dating the formal publication of online-before-print articles. Such a tactic, I argued, would show up in detectable pattern: consistently smaller December issues and grossly inflated January issues.
After getting a tip from a colleague that a journal editor may be engaging in this type of behavior — as well as coercing authors to cite the journal — I decided to do a little data digging. For a frame of reference, this is a monthly journal in biological engineering owned and published by a commercial publisher.
I used Web of Science to download publication data for 7,344 original articles published between 2000 and 2011. My analysis looks at the number of articles published in each monthly issue, for each year, over 12 years. If there has been a consistent pattern of issue-loading, analyzing these data should be able to detect it.
Indeed, controlling for the growth of the journal over time, January issues were 68% larger (95% C.I. 25% to 126%, p=0.0007), on average, than all other months. In fact, the January, 2011 issue was the largest issue to date, publishing 295 research articles. January is represented by the red line in the regression plot below.
However, there is no evidence that December issues were any smaller. These issues were just 0.01% smaller (95% C.I. -26% to 34%, p=0.992), on average, than all other issues. August issues (mustard line) were generally the leanest issues of the year, publishing the fewest articles, which may simply reflect the cyclical nature of manuscript flow to the journal.
Last, while this journal is growing over time, there is no evidence that December issues are getting relatively smaller or that January issues are getting relatively larger. There is no detectable interaction effect in our regression model between these months and YEAR, which is what one would expect if editors found that they were on to a good thing.
Using one suspect journal to disprove a conjecture is hardly overwhelming evidence. More importantly, my analysis takes no account of the flow of manuscripts into the system and the delay between acceptance and publication. If we are to find a smoking gun, we need evidence that editors are actively shifting publication dates for the purposes of gaining citation impact and not for other reasons. As one managing editor expressed to me recently:
It’s a topic of discussion. Whether or not it happens, the editors are worried about the perception that it’s taking place.