F1000 Research is a new initiative on the science publishing landscape – something like an article-hosting service with post-publication comments. Papers that not approved by enough commenters are rejected, in that they are no longer publicly visible. The article processing charges (APCs) vary between $250 and $1000 dollars per article, with the price depending on article length.
PubMed is sufficiently impressed that they include approved F1000 Research articles in their searches, but just how thoroughly are their papers being vetted? Here, I compare a few features of the post publication review process at F1000 Research with pre-publication peer review at four high-ranking medical journals.
Most strikingly, commenters at F1000 Research made very little effort to improve the paper — F1000 Research comments were typically short and positive, whereas pre-publication reviews at the medical journals were longer and more negative. The figure shows the average word count by recommendation for the first review in 25 recent submissions to the medical journals (total = 100), compared to 25 from F1000 Research.
Of the 25 reviews in F1000 Research, 18 were under 200 words (four had zero words), and 21 (84%) were positive. The average length was 254 words. By contrast, the average length for the medical journals was 464 words, and only 42% were positive.
Maybe the F1000 Research commenters were inexperienced, and did not understand how to write a full-length review? Not so. The average h-index of these 25 F1000 Research commenters is 24.7, which implies a long and distinguished publication record. It appears that the journal is actively encouraging the F1000 Faculty Members to provide reviews, but even when these 12 people are excluded, the average h-index of the F1000 Research commenters is still 18.7. All of these commenters know how to review a paper in the traditional pre-publication setting.
These 25 F1000 Research papers are unlikely to be perfect, and some probably contain major flaws. These commenters would very likely have asked for more improvements had they been writing pre-publication reviews. Why did so many of them just click “Approve” and wave the paper through mostly unchanged?
First, the editorial policy of F1000 Research incentivizes short, positive comments:
Referees are asked to provide a referee status for the article (“Approved”, “Approved with Reservations” or “Not Approved”) and to provide comments to support their views for the latter two statuses (and optionally for the “Approved” status).
Approving a paper is thus the path of least resistance.
Another problem facing commenters is motivation – the article’s horse has already bolted into the public domain. Attempting to shut the stable door by detailing the flaws isn’t worthwhile, and the authors are unlikely to make any substantial changes in the light of their comments (Gøtzche et al found a similar problem for Rapid Responses at BMJ.
In addition, writing a negative report may upset the authors. Nobody would take an anonymous comment seriously, so the comments have to be signed. The choices for a flawed paper are thus be nice and give it an approval, or write up an honest evaluation and risk a fight. This design pressures commenters to keep their comments positive.
In pre-publication review, you know that your comments will be evaluated by a senior figure in your field (the editor), and their opinion about you can affect your career. There’s also real risk that a weak review will get shot down in the decision letter: “I have chosen to ignore the superficial review of referee 1, and instead have based my decision on the more comprehensive and critical comments of reviewers 2 and 3”. Ouch.
There’s no equivalent of a deciding editor at F1000 Research (three “Approvals” are all that is required), so while there’s a risk that someone will dismiss your review, the probability that it’s someone whose opinion you care about is much lower. It’s also much less likely that a short, positive assessment for F1000 Research will be the odd-one-out, as long, negative reviews are so much less common. Lastly, in pre-publication peer review, the editor will be reluctant to recommend acceptance if he doesn’t think the paper has been thoroughly examined, but this quality control step is absent at F1000 Research.
Another crucial point is how the prospect of rigorous peer review affects the motivation of authors. Knowing that their paper is likely to be scrutinized by experts in their field drives researchers to eliminate as many errors as possible. This motivation may be absent for authors preparing to submit to F1000 Research, as it is readily apparent that most comments are short and positive. The combination of a lack of care from authors and lack of scrutiny from reviewers may allow badly flawed papers to get the stamp of approval.
More broadly, does it actually matter if online commenting doesn’t replicate pre-publication review? If the papers are getting “approval” from experts in the field, does it matter that the papers still contain errors that might have otherwise been caught?
Reading and reviewing are not the same activity. Reviewers typically start with the position “this paper is flawed and should not be published” and expect the authors to convince them otherwise. Reading a paper is different, and is more focused on how the questions and conclusions relate to your own research. The time commitment involved is also very different — a quick poll around some postdocs and senior PhD students found people spend 6-10 hours reviewing a paper, but only 30-90 minutes reading an article before deciding whether to cite it in their own work.
When researchers are reading a paper, they work from the assumption that someone has gone carefully through it and helped the authors deal with a significant proportion of the problems. This last assumption cannot safely be made for F1000 Research papers, because there is no way to know whether the “Approve” recommendations mean there are no errors in the paper, or that the commenters did not look very hard.
Responsible researchers should therefore approach every paper from F1000 Research as if it has never been through peer review, and before using it in their research they should essentially review it themselves. This imposes a new burden on the community — rather than take up the time of two or three researchers to review the paper pre-publication, everyone has to spend more time evaluating it afterwards.
There may also be an irony here. Authors opting to use F1000 Research to publish as fast as possible may find that uptake comes slower while everyone else finds time to check over the article. Lingering doubts over reliability may also mean that many ignore the work altogether.
There are a few tweaks that F1000 Research could make to ease these concerns. First, their editorial board lists many exemplary researchers, so why not use their expertise in a more “editorial” rather than a “commenter” role? People giving casual approval to flawed papers would then have much more reputation at stake.
Second, the editorial policy of making “approval” the path of least resistance combined with collecting APCs for approved papers means that F1000 Research flirts with predatory OA status. This is unfortunate and unnecessary. Why not switch to a submission fee model? This makes particular sense for F1000 Research as most of their services are provided pre-approval anyway; it would crucially detach approval decisions from financial reward.
A closing thought — the progress of science does not depend on how many articles are published and how quickly. Instead, progress depends on how well each publication serves as a solid base for future work. Gratifying authors with positive comments and instant publication may be a successful business model, but we will all lose out if the ivory towers are increasingly built with sand.
* The journals contributing to this analysis were the New England Journal of Medicine, CHEST, Radiology, and the Journal of Bone & Joint Surgery. Thanks to the staffs of those journals for their help, to Kent Anderson and Elyse Mitchell for their assistance in collecting the data, and to Arianne Albert for the figure.