Authority, Authors, Ethics, Experimentation, Marketing, Peer Review, Reading, Research, Social Role, Tools

PubMed and F1000 Research — Unclear Standards Applied Unevenly

PubMed: Now with 20 million citations

PubMed: Now with 20 million citations (Photo credit: dullhunk)

What is PubMed? Ask most authors or editors, and they’re likely to say that it’s another version of MEDLINE. That is, it’s an index of biomedical journals that is difficult to get into because it has high standards for acceptance. However, as we know, PubMed is not equivalent to MEDLINE, nor has it been for a long while. PubMed contains MEDLINE records, but also contains hundreds of thousands of other records. There are at least two pathways for admission — one that remains difficult and time-consuming and flows through the MEDLINE journal selection process, and another that is not only easy, but sometimes is actively facilitated by the NCBI. That is, if you’re willing to deposit open access (OA) content in PubMed Central (PMC), you might get into PubMed on an expedited basis via PMC — especially if you’re viewed favorably by NCBI personnel. It’s a form of leniency granted to publishers with a business model the NCBI implicitly endorses.

This secondary route into PubMed has apparently become even more lenient with the admission of reports from a non-journal, F1000 Research. F1000 Research seems to avoid calling itself a journal. It calls itself an “alternative” and a “website” and a “publishing program.” It has articles, editorial practices, and a form of peer-review. But it refuses to call itself a journal, except in one instance, which seems more of an error than an intentional usage. In some communications, it juxtaposes itself against journals.

As far as peer-review goes, F1000 Research uses what they call “open” post-publication peer-review, but which is really a cynical and confusing mélange of incomplete editorial practices. Papers emerging from this approach to peer-review won’t be indexed in PubMed (via the PMC route) unless they’ve passed peer-review — although what “passed” means isn’t entirely clear or uniformly applied, as we’ll see. Nothing’s been indexed yet — all we have is an announcement from F1000 about the acceptance of F1000 Research reports into PubMed via PMC deposits.

F1000 Research is focused on very different criteria than you might expect — namely, speed and citable objects. According to their announcement:

With our record for publication time of 30 hours and our record for receipt of referee reports of 24 hours, you could find your article listed in PubMed in under 3 days from submission!  Could prove rather handy if you need something quick for those looming grant deadlines or you need to ensure you don’t get scooped.

F1000 Research is explicitly providing researchers with a shortcut to cram more citations into their CVs in a pinch, while eliding the fact that they are not publishing in a journal. Being indexed is part of projecting a misleading image. They don’t care about quality or relevance, don’t have an editor, don’t call themselves a journal — they just provide authors with an academic chit as quickly as possible. That’s cynical, and NCBI has agreed to play this game. Worse, F1000 Research calls what it’s doing “publication,” when in fact, it’s much more confusing than that.

For instance, there are articles in F1000 Research which have been effectively rejected — that is, disapproved by two peer-reviewers and approved by zero. (There are at least three such articles out of 84, or about 3% to-date.) Yet, these rejected articles continue to be published on the F1000 Research site. What journal publishes articles its peer-reviewers have rejected? It’s a contradiction, and further evidence that F1000 Research isn’t operating as a journal. It’s also problematic for the authors, whose work is being publicly rejected, and for legitimate journals, which may consider subsequent publication of rejected but published works as duplicate publication.

At its base, F1000 Research is not a journal but an open editorial office. It’s just like seeing into an editorial review process, which muddies the waters for everyone involved. They are essentially equating the unrestrained aspect of their review process with “publication,” but how can you publish papers your own reviewers have rejected or papers that haven’t been reviewed at all? Maybe publication comes later, after acceptance, which is when indexing occurs? That seems to be their rationalization.

Unfortunately, the publication line is not very bright at F1000 Research.

In its author guidelines, F1000 Research elaborates a bit on how an article becomes indexed:

Once your article receives two Approved statuses, or two Approved with Reservations statuses and one Approved status, your article will be indexed — currently in PubMed, PubMed Central, Scopus, Embase, Google Scholar, CrossRef and the British Library — and the status of your article will change to ‘Indexed’.

F1000 Research doesn’t seem to be enforcing these rules precisely. For instance, one article was listed as “indexed” despite receiving two Approved with Reservations statuses and no Approved status. Then, the same situation occurred with another article. And another. One article was indexed despite receiving one Approved, one Approved with Reservations, and one Not Approved. Another was indexed despite receiving one Not Approved and two Approved with Reservations.

How are peer-reviewers selected at F1000 Research? Mainly, by authors. After all, this is an open editorial office:

As an author you will be asked to identify 5 potential referees, primarily from the F1000Research Editorial Board. If necessary, you may select others of suitable standing.

There’s an obvious problem with this — a real potential for reviewer bias, even with the caveats F1000 Research tries to put on the process. Many journals allow authors to suggest peer-reviewers, but few solicit it explicitly limit their selections to author recommendations like this. Depending upon authors for reviewer suggestions isn’t a rigorous approach to objective peer-review. This is another indication that F1000 Research is quite right to avoid labeling itself a journal.

What do these reviews look like? They fall short of peer-reviews I’ve seen. In scanning F1000 Research, I’ve seen approvals without comment; approvals with scant comments; rejections with scant comments; and so forth. This seems like inadequate peer-review to me, and only underscores that you can’t achieve legitimacy just by saying something is peer-reviewed — how you manage and interpret peer-review really matters, which is why editorial review exists. Peer-review is a tool, not a solution.

Indexing also creates a potential problem for F1000 Research — after all, their value proposition is based on publication, which at first blush seems like making something public, but which for them means registration of the published works with the major indexing services. If F1000 Research generates a ghetto of unreviewed or rejected papers, what will that mean for those authors and for F1000 Research? One paper from November is currently unreviewed. Is that a refund waiting to be processed? What obligation does F1000 Research have to its authors? And what obligation to readers? Rejected or unreviewed papers can still be downloaded. What kind of filter is F1000 Research?

And what criteria apply to PubMed these days, if any? PubMed, PubMed Central, and MEDLINE are confusing to practitioners in many fields, including editors and authors. Now that it’s clear that PubMed is actively courting scale through OA (see its new slogan in the artwork above), even to the point of indexing a non-journal that is publishing articles its own peer-reviewers have rejected.

What standards does PubMed represent?

Ultimately, PubMed is courting brand problems. A brand is a promise. PubMed’s promise has long been understood as providing an objective, high-standard index of biomedical journals. It is no longer providing that. It is now a compromised database with a mixed bag of standards and content sources, with rules that are unevenly enforced and possibly unclear to even those operationalizing them.

What is PubMed? What is F1000 Research? Those are both very good questions.

Enhanced by Zemanta

About Kent Anderson

I am the CEO/Publisher of the STRIATUS/JBJS, Inc., the home of the Journal of Bone & Joint Surgery, JBJS Case Connector, JBJS Reviews, JBJS Essential Surgical Techniques, the JBJS Recertification Course, PRE-val, and SocialCite. Prior to this, I was an executive at the New England Journal of Medicine. I also was Director of Medical Journals at the American Academy of Pediatrics.

Discussion

30 thoughts on “PubMed and F1000 Research — Unclear Standards Applied Unevenly

  1. “For instance, there are articles in F1000 Research which have been effectively rejected — that is, disapproved by two peer-reviewers and approved by zero. (There are at least three such articles out of 84, or about 3% to-date.) Yet, these rejected articles continue to be published on the F1000 Research site.”

    I can see this being a real problem. Most journals probably won’t consider an article that is the same as or similar to a ‘rejected’ article that stays posted on F1000 Research, viewing this as duplicate publication. What if it isn’t declared by authors and not picked up by journals on submission or during review? Not all authors are aware of what is/is not allowed, what constitutes prior publication – which is complex and changing area – and not all journals do text similarity checks.

    Also, what are the implications of the ‘rejected’ articles bearing the CrossMark logo?

    Posted by Irene Hames | Jan 15, 2013, 10:08 am
    • It’s really unclear what one’s options are if one’s paper is rejected in a system like this. Their instructions to authors suggest you can revise the paper and post a new version, but there is no information on whether that new version will be newly reviewed, and whether it is possible to change the status on the paper from rejected to accepted. Nor is there any information on how many times a paper can go through the review process. Are reviewers willing to review and re-review the same paper hundreds of times or are there limits? Are there additional fees charged for additional rounds of peer review?

      Can you retract a rejected paper, then revise it and submit it somewhere else? Would an editor still consider that “previously published” work?

      Posted by David Crotty | Jan 15, 2013, 10:50 am
      • I’ve tweeted Ed Pentz for feedback on the CrossMark issue and also F1000 Research to ask if a ‘not-approved’ article can be submitted elsewhere. I’ll post what they say if they don’t come here direct to comment.

        Posted by Irene Hames | Jan 15, 2013, 11:24 am
        • Thanks. You’ve raised an interesting question.

          Posted by Kent Anderson | Jan 15, 2013, 11:25 am
          • I asked: @epentz implications of not-approved articles with CrossMark logo being sub+pub elsewhere? See http://bit.ly/W8bVrK @F1000Research can they be?

            Replies from F1000Research:

            @irenehames @epentz No, once published then that is it. It should help to stop poor quality science being further disseminated 1/2

            @irenehames @epentz If the work is just ahead of its time then others can say this in comments & other referees can be added at any time 2/2

            Posted by Irene Hames | Jan 15, 2013, 12:42 pm
            • If that is indeed accurate then it seems quite a gamble for authors, having only one shot to get through peer review. For most of the journals I work with, the majority of papers go through at least one revision before acceptance.

              Posted by David Crotty | Jan 15, 2013, 1:08 pm
            • Reply from Ed Pentz via twitter: “no implications for CrossMark – it’s published.”

              Posted by Irene Hames | Jan 17, 2013, 8:37 am
  2. Any self-respectuing scientist doing a lit search would be able to tell peer-reviewed from non-peer reviewed articles even if they were on PubMed. I think that the more inclusive the PubMed search engine is, the more useful it becomes and I have no problems having it include F1000 articles. In fields such as physics it is common to use ArXiv as a pre-publication article repository and I think F1000 is serving a similar function. I don’t think anyone considers having a journal indexed at PubMed particularly hard or prestigious. They simply have the minimum filters to keep out vanity press journals.

    Posted by namnezia | Jan 15, 2013, 10:44 am
    • An interesting final statement. Why isn’t F1000 Research a “vanity press”? It’s not even claiming to be a journal, and the authors have significant input into the selection of reviewers, and pay for publication.

      As for PubMed’s reputation, only a small percentage of working scientists know that it has devolved into something rather simple with minimal standards for inclusion, and a pro-OA bias at that.

      Posted by Kent Anderson | Jan 15, 2013, 10:52 am
      • I’ll second that when I was a journal editor, I frequently heard from authors that in order for their paper to “count” toward career advancement, it had to be in a journal that had an impact factor and that was indexed in PubMed.

        Posted by David Crotty | Jan 15, 2013, 10:56 am
      • F1000 is not vanity press in that it is trying to introduce a model of open peer review (much like ArXiv) where the article is reviewed post-publication and the review process is transparent and ongoing, letting the article be judged by the broader science community. This is very different than vanity press, and potentially much more stringent in the end than lots of crappy journals that have “made the cut” to be indexed in PubMed. I disagree, I think most working scientists know very well that PubMed is simply a useful tool for searching the biomedical literature, no one would boast of being published “PubMed Indexed Journals” and if anything it hasn’t devolved, it has evolved into an increasingly useful and free tool. That it is embracing a push for more open access publishing is something that should be celebrated, not derided.
        Finally, in the 20 years that I’ve been publishing articles, the editors ALWAYS have requested names of potential reviewers, both in high profile and second tier journals.

        Posted by namnezia | Jan 15, 2013, 11:04 am
        • Why work for indexing in PubMed then? If it’s just a pre-print repository, why all the trappings of a journal?

          Requesting reviewer names as an option is far different from pointing authors to a set of reviewers and requesting five from that pool, and having no editor atop the publication to adjudicate. That gives authors a lot more control.

          However, the more worrisome part of F1000 Research is that it’s not following its own criteria — indexing articles that don’t meet its own peer-review criteria.

          Posted by Kent Anderson | Jan 15, 2013, 11:17 am
          • They want to be indexed in PubMed because it is pretty much the only tool that biomedical researchers use to search the existing literature. Research that isn’t indexed in PubMed (but that from a topic standpoint could be) is simply not going to be found, and therefore cited, as much as research that is. From an author’s standpoint, publishing in a journal that wouldn’t allow your paper to come up in a PubMed search pretty much guarantees that your paper disappears into the void.

            Posted by Dr Becca | Jan 15, 2013, 4:45 pm
        • F1000 Research could indeed be ‘much more stringent’, but it currently is not. Most of the approvals consist of a sentence or two, and there is no guarantee that the ‘reviewer’ has read anything more than the title. Editors at even low tier journals wouldn’t publish a paper on the back of such weak and cursory reviews.

          Posted by Tim Vines | Jan 15, 2013, 1:05 pm
          • There’s also a critical distinction between arXiv and F1000 Research. ArXiv has no online comments and relies on private communication between authors and readers, which allows for a) a much more frank discussion and b) an opportunity for authors to assess the credentials of the person giving the feedback. F1000 research uses mainly online comments, which tend to be positive (to avoid flamewars), and, since it takes less effort to post online than write a private letter, can come from more or less anyone. F1000 Res will probably not be able to emulate the rigorous vetting process of arXiv, particularly if it serves as the manuscript’s final destination and not as a preprint server.

            Posted by Tim Vines | Jan 15, 2013, 5:31 pm
    • PubMed is widely promoted to, and used, by consumers who are not “self-respecting scientists”. It is simply naive to ignore the brand value and credibility that PubMed indexing brings to content – especially in the eyes of consumers who are gaining increased access to the original literature thanks to OA policies.

      Posted by Richard Wynne | Jan 15, 2013, 11:56 am
  3. “Many journals allow authors to suggest peer-reviewers, but few solicit it explicitly.” Not sure where you’re getting this– I’ve never submitted to a journal that DIDN’T ask me explicitly to provide at least 3 names of potential peer reviewers (and no, I do not publish exclusively in OA journals). My impression is that editors will usually go with one of the suggested, but the other 2 or 3 are recruited independently by the editor.

    I’ve been using PubMed for over a decade, and have never thought of it as needing “standards” or a “brand.” It’s a resource for biomedical scientists to find published research based on keyword or author searches, and that’s it. Its job is not to be some sort of endorsement of the scientific integrity or value of its content–that’s for the scientists reading the papers to judge.

    Posted by Dr Becca | Jan 15, 2013, 10:53 am
    • In the context the sentence occurs, I was trying to point out how specific their approach is — five recommended reviewers from the F1000 Research advisors. This is different from most journals, who allow authors to suggest reviewers, then make their own determination about who to use. It’s not clear how F1000 Research handles it, frankly, and they seem to give authors much more input than traditional journals. After all, there is no editor at the helm sorting things out.

      Posted by Kent Anderson | Jan 15, 2013, 11:00 am
    • Medline’s criteria for acceptance can be found here:
      https://www.nlm.nih.gov/pubs/factsheets/jsel.html

      The decision whether or not to index a journal for this service is an important one and is made by the Director of the National Library of Medicine, based on considerations of both scientific policy and scientific quality….

      Quality of content: Scientific merit of a journal’s content is the primary consideration in selecting journals for indexing. The validity, importance, originality, and contribution to the coverage of the field of the overall contents of each title are the key factors considered in recommending a title for indexing, whatever the intended purpose and audience…

      Quality of editorial work: The journal should demonstrate features that contribute to the objectivity, credibility, and quality of its contents. These features may include information about the methods of selecting articles, especially on the explicit process of external peer review; statements indicating adherence to ethical guidelines; evidence that authors have disclosed financial conflicts of interest; timely correction of errata; explicit responsible retractions as appropriate; and opportunity for comments and dissenting opinion…

      Production quality: Quality of the layout, printing, graphics, and illustrations are all considered in assessing a journal…

      Posted by David Crotty | Jan 15, 2013, 11:07 am
  4. To think you get this rapid pub schedule, cited in PubMed a free review from your friends and all for only $250 to $1,000. Not a bad way for someone to make a living!

    And so it goes – garbage in garbage out!

    Posted by Harvey Kane | Jan 15, 2013, 12:07 pm
  5. Am curious about this statement:- “Now that it’s clear that PubMed is actively courting scale through OA (see it’s new slogan in the artwork above)” which makes no sense to me.

    Posted by Graham Steel (@McDawg) | Jan 15, 2013, 2:03 pm
    • When you boast quantity (size), you are playing a different game than a quality (filter) game. When you create an easier path for indexing which is predicated specifically on OA compliance, you are courting that scale through OA. That’s the logic I was using.

      Posted by Kent Anderson | Jan 15, 2013, 2:19 pm
  6. I find all of this shocking albeit not surprising.

    I am currently reviewing Medline listings of accepted journals in the past 5 years. With my co-authors we will submit an abstract to the World Congress on Peer Review and Biomedical Publication. Our study question will focus on bias in Medline selection that hurts non English language journals, “local” journals, and emerging country journals that do not belong to the big commercial publishers.

    Unfortunately, we cannot ignore the fact that being in “PubMed” is rated highly by academic institutions – and other indexing systems – who couldn’t care less about how XX or YY journal got into PubMed. Furthermore, all systematic reviews include PubMed/MEDLINE as the starting point, adding few other first-world data bases at best.

    As someone who runs a journal that was recently rejected by Medline, and who cannot take her journal to PubMed Central since they only accept English language journals, I feel great disencouragement when others find easy shortcuts with dubious quality editorial projects.

    Maybe none of our hard work to provide Latin America with a high quality OA and totally free biomedical journal is just quijotesque.

    Posted by Vivienne Bachelet (@V_Bachelet) | Jan 15, 2013, 2:04 pm
  7. F1000Research is a refreshing change in publishing, a change that should be welcomed. Like anything else that is new, it has areas that need improvement but there is no reason to dismiss this effort. The fact that they require the authors to submit their raw data is welcoming. As an active researcher and reviewer I am very pleased to see this platform.

    Posted by Khosrow Kashfi | Feb 6, 2013, 1:41 pm

Trackbacks/Pingbacks

  1. Pingback: Mulitgrain Discussion: PubMed, F1000 Research, Peer Review and the question of standards | Against-the-Grain.com - Jan 21, 2013

  2. Pingback: Pulling the Wool Over Their Eyes — The PubMed National Advisory Committee and Conflicts of Interest « The Scholarly Kitchen - Feb 8, 2013

Side Dishes by Stewart Wills

Find Posts by Category

Find Posts by Date

January 2013
S M T W T F S
« Dec   Feb »
 12345
6789101112
13141516171819
20212223242526
2728293031  

The Scholarly Kitchen on Twitter

SSP_LOGO
The mission of the Society for Scholarly Publishing (SSP) is "[t]o advance scholarly publishing and communication, and the professional development of its members through education, collaboration, and networking." SSP established The Scholarly Kitchen blog in February 2008 to keep SSP members and interested parties aware of new developments in publishing.
......................................
The Scholarly Kitchen is a moderated and independent blog. Opinions on The Scholarly Kitchen are those of the authors. They are not necessarily those held by the Society for Scholarly Publishing nor by their respective employers.
Follow

Get every new post delivered to your Inbox.

Join 13,053 other followers

%d bloggers like this: