A hypothesis without data is valuable if it is based on theory and sound reasoning. We call these conjectures.
Last Monday, I explored a conjecture proposed by Frank Krell, that journals are manipulating their impact factor by post-dating the formal publication of online-before-print articles. Such a tactic, I argued, would show up in detectable pattern: consistently smaller December issues and grossly inflated January issues.
After getting a tip from a colleague that a journal editor may be engaging in this type of behavior — as well as coercing authors to cite the journal — I decided to do a little data digging. For a frame of reference, this is a monthly journal in biological engineering owned and published by a commercial publisher.
I used Web of Science to download publication data for 7,344 original articles published between 2000 and 2011. My analysis looks at the number of articles published in each monthly issue, for each year, over 12 years. If there has been a consistent pattern of issue-loading, analyzing these data should be able to detect it.
Indeed, controlling for the growth of the journal over time, January issues were 68% larger (95% C.I. 25% to 126%, p=0.0007), on average, than all other months. In fact, the January, 2011 issue was the largest issue to date, publishing 295 research articles. January is represented by the red line in the regression plot below.
However, there is no evidence that December issues were any smaller. These issues were just 0.01% smaller (95% C.I. -26% to 34%, p=0.992), on average, than all other issues. August issues (mustard line) were generally the leanest issues of the year, publishing the fewest articles, which may simply reflect the cyclical nature of manuscript flow to the journal.
Last, while this journal is growing over time, there is no evidence that December issues are getting relatively smaller or that January issues are getting relatively larger. There is no detectable interaction effect in our regression model between these months and YEAR, which is what one would expect if editors found that they were on to a good thing.
Using one suspect journal to disprove a conjecture is hardly overwhelming evidence. More importantly, my analysis takes no account of the flow of manuscripts into the system and the delay between acceptance and publication. If we are to find a smoking gun, we need evidence that editors are actively shifting publication dates for the purposes of gaining citation impact and not for other reasons. As one managing editor expressed to me recently:
It’s a topic of discussion. Whether or not it happens, the editors are worried about the perception that it’s taking place.
Discussion
5 Thoughts on "The (Post) Dating Game — Assembling the Evidence"
You might also hypothesise that annual page budgets restrict the number of articles publishable towards the end of a year. A January issue would fall within a new budget year and might give an editor the chance to put in some of the articles he or she had been forced to shelve in November and December. To distinguish this effect from the impact factor hypothesis, you might reasonably suggest that the budget effect would be more noticeable in growing journals – a larger page budget the following year would be more conducive to holding over articles than a constant page budget which would require a more disciplined approach. The impact factor hypothesis shouldn’t be affected by whether the journal is growing in size.
To really model effects, it would seem to me at first glance that one should look at all relevant dates: submitted, [revised,] accepted, on-line, and in-print. It would also seem relevant to consider the calendar effects such as the Christmas vacation schedule at teaching institutions and summer vacations in August.
Looking at submissions by week, I would expect lower than average submissions during vacations and perhaps higher than average submissions immediately after, especially after the Christmas vacation. Looking at the dependence of average time-to-acceptance on the submission week, I would expect a lengthening of the time when the “referee” period included a vacation. It would be interesting to see what effect, if any, vacations have on the time between acceptance and on-line. It seems to me that variations in the time between on-line and in-print would be most subject to editorial control in terms of page budgets, topical issues, related papers together, or other factors of interest to the editors.
If there would be a way to “normalize” the time scales for different journals (i.e., adjust for journal and/or field differences in the average time delays), then it might be possible to construct a “standard” model taking standard cyclic factors (vacations, etc.) into account. Then it might be possible to detect some journal(s) that have a significant departure from the “standard” model and question whether the departure is a result of an attempt to manipulate the impact factor.
I agree. As an editor I noticed a real cyclical pattern to submissions. A big drop off in submissions through the summer on into the fall with a bolus right before Christmas and then dropping off after the holidays and building up to another peak in the spring. This might differ by field/journal etc. but definitely something you might want to consider as a possible alternate explanation.
It’s also quite possible that articles were delayed into the next year if the publisher operated on a calendar-year budget cycle and wanted a smaller printing bill in the current fiscal year.
It’s not just a smaller printing bill. By delaying, or bringing forward, publication of an issue a publisher can delay, or bring forward, the earnings of that particular issue. If a journal publishes 12 issues per year and brings in $120,000 in susbcription revenue, each issue earns $10,000 when published. Having flexibility about when you publish these issues allow you to manage your earnings from one period to the next, catching up when you are behind, or holding steady when you don’t want to exceed your target.
