Authority, Authors, Business Models, Controversial Topics, Ethics, Experimentation, Marketing, Peer Review, Research

Seeking Acceptance at F1000 Research — Early Problems With Identity and Outsourced Authority

English: Car upside down. Suomi: Auto katollaa...

English: Car upside down. Suomi: Auto katollaan tennishallin parkkipaikalla Porissa. (Photo credit: Wikipedia)

F1000 Research seems to delight in being surreal — publication before review, authority generated extrinsically instead of intrinsically, and “reviews” consisting of zero words of feedback.

It’s a topsy-turvy publishing model.

Yet, F1000 Research has been accepted into the PubMed index via PubMed Central. It has also since redefined itself as a mega-journal, after initially saying expressly it was not a journal. Its search for identity may be linked to its inability to assert its own authority.

How did F1000 Research get indexed in PubMed? Let’s look at what I recently obtained via the ongoing and expanded Freedom of Information Act (FOIA) request filed with the National Institutes of Health (NIH). The materials reveal how PMC changed its role relative to journals in order to accommodate the F1000 Research model. They also lay bare the contradictory mechanics operating within the F1000 Research model itself.

F1000 Research applied for inclusion in PMC in October 2012. Because F1000 Research is an open access (OA) journal, inclusion in PMC also grants it indexing in PubMed (but not MEDLINE — owing to the rampant confusion between PubMed and MEDLINE, however, this distinction is for specialists only). It was granted inclusion in December 2012, with some stipulations, as this email from the NLM to Rebecca Lawrence of F1000 on December 10, 2012, shows:

Dear Rebecca,

NLM’s Selections group has approved F1000 Research. After careful consideration, NLM has decided that PMC will only accept an article after it has:

*  Two unconditional Approves,

*  Or, two Approved with reservations + one unconditional Approve

This is [a] new model and we need a standard that can be applied broadly to other journals. The second condition addresses concerns that arose during the review process that “Approved with Reservations” is not as strong as an unconditional Approval.

In the email responding to news of PMC’s approval of F1000 Research for inclusion and indexing in PubMed, Lawrence’s response probes gently to see if PMC would consider keeping the previous F1000 Research standard:

This is great news; thank you for the confirmation and we accept your proposal below. One clarification question — in the case of one Approved with Reservations and one unconditional Approved, is that acceptable as well?

In response, PMC reiterates its standards:

To clarify, an article with only one Approved with Reservations + one unconditional Approval would not be acceptable for deposit in PMC.

Notice that now PMC is saying that their approval is for deposit in PMC, not indexing in PubMed. It’s a small point, but the two are so conflated that the elision is natural.

This “Approved with Reservations” status bears further scrutiny. What exactly is it? According to F1000 Research’s PMC application, submitted on October 3, 2012, by Rebecca Lawrence:

. . .

Referees are asked to:

*  Provide a referee status for the article of ‘Approved’ (i.e. this is good science); ‘Approved with Reservations’ (i.e. it is still science but there are some reservations about the study); or ‘Not Approved’ (i.e. this is not science) – this status is then immediately displayed against the article and the referee’s name.

We’ve seen criteria for acceptance expand a good deal with things like “methodologically sound” or “scientifically sound” for some of the mega-journals. These standards eschew novelty or interest (I’m going to be writing my paper about coin flipping any day now, because it will likely be methodologically sound). But this new set of definitions gets more basic yet — no longer do the methods or the science need to be “sound” according to F1000 Research. In this new vision, the methods only need to amount to “science.” Any science. Boring. Methodologically flawed. Repetitive. Irrelevant. Just as long as it’s science.

It’s part of the upside-down world of F1000 Research that “Approved with Reservations” counts as an approval status. This is the opposite of what a similar status would be in a pre-publication review system. I can’t say that emphatically enough — it is the op-oh-sit! But F1000 Research describes it as a “positive referee status” in its application:

What we have agreed with other indexers such as Scopus and Embase is that once the article receives two positive referee statuses, it is then sent to them for indexing and we would like to propose the same system for PubMed Central.

Another email from October 24, 2012, confirms this backwards point of view.

For journals using pre-publication peer review, the status of “Approved with Reservations” may not seem all that unusual. It’s a signal to an author that the paper is very good, but needs some fine-tuning or more analysis or rewording of key conclusions before it gains final acceptance. However, the paper is not accepted and certainly not published, and the status is not a “positive referee status” but merely a less-negative referee status. But it is still negative. It is a form of conditional rejection. Retaining the leverage of publication all but guarantees that the author will make the minor modifications the editors and reviewers require.

In its backwards model, F1000 Research has already published the paper, so “Approved with Reservations” is a toothless de facto approval. Yes, the reservations are included in the online review, but readers have to sort through them, and they are not often terribly detailed. In any event, the problem runs deeper.

Because the papers are published before they are reviewed, they can’t be unpublished. PMC tried to see what might happen in this case in October 2012, in a long list of questions:

5. What happens to the status of the article if two Approvals are registered, followed later by two Rejections?

The answer is a bit evasive and technically incorrect:

5. In that scenario, the indexed status of the article would be maintained. The only time this might change is in the case of a retraction, withdrawal, replacement or removal – see http://f1000research.com/crossmark-policy/ for further details on those scenarios.

That URL (and even one corrected to add the www.) leads to a 404 error, so I don’t know how PMC could have checked for further details. But that’s not all that’s wrong with this response. First, an article retraction or withdrawal usually only changes the status of its entry in an index. As the NLM states:

NLM does not remove the citation for a retracted article, but updates the citation to indicate it has been retracted, and links the original citation to the citation for the published retraction notice.

Claiming that an article that’s retracted would no longer be indexed is incorrect.

Lawrence doesn’t answer the question directly, and it’s an important one. In the F1000 Research system, an article is published, then reviewed. The reviews can change the indexing status of an article, but only once. According to the agreement made by F1000 Research and PMC, an article can be indexed with two “Approved with Reservations” and one “Approved.” That article might also then be revised to deal with the reservations found in the two “Approved with Reservations,” despite the journal having no real leverage over authors at this point. Those revisions could lead to two “Not Approved” statuses on Version 2. What then? The paper has been published and indexed. It can’t be unpublished. And apparently, it can’t be unindexed. Does it have to be retracted then? And what if further revisions improve the paper so that Version 3 is exactly what the reviewers ordered? Is it unretracted?

As we’ll see below, there may never be a final paper in F1000 Research, only temporal versions.

This issue came up during the back and forth between F1000 Research and PMC, but the exchange is brief and unsatisfying. First, PMC’s question:

1. Does a Referee Report link to a specific version of the article, or to all versions of the article?

Lawrence’s response seems at first glance to make sense, but falls apart when you think about it:

A referee report is linked to a version. However, for say Version 2 of an article, all referee reports to both V2 and V1 are shown at the bottom of the article, whereas for Version 1 of the article, you only see Version 1 referee reports. The referee status of the V2 article is the most recent referee status provided by each referee (e.g. for V1, say Ref 1 says tick; Ref 2 says ?; Ref 3 says X; and then in V2, Ref 3 now says tick. So the Status on V2 is tick, ?, tick).

There are strong assumptions in this scenario — that Version 2 doesn’t deviate in important ways from Version 1, such that the initial reviewers’ “ticks” are no longer valid; that Version 2 isn’t re-reviewed by the first reviewers as well, and doesn’t end up with two “?” and one “tick.” More significantly, the scenario is never considered in which an author receives two Approvals and is indexed then goes back in, modifies the paper significantly, carries the two Approvals forward into a version the reviewers would effectively never have reviewed as it deviated so much from the initial paper, and then sits back and lets biased information permeate the landscape. It would be difficult to detect the changes and impossible to know that the reviewers had not approved of the latest version. And don’t tell me this is far-fetched — we’ve seen a lot of misbehavior over the past few years among authors.

There are a lot of problems with the F1000 Research model.

Another problem was revealed when PMC forced F1000 Research to change their standards mid-stream. This has created problems for authors who initially met the F1000 Research standards for having their articles indexed. As a blog post on the F1000 Research site noted in February:

In 2012, we agreed with Scopus and others that an article would be indexed as long as it had obtained any two of our ‘Approved’ or ‘Approved with Reservations’ statuses.  At the start of 2013, NLM agreed to index our content but they requested slightly more stringent criteria: either two ‘Approved’ statuses, or two ‘Approved with Reservations’ statuses plus one ‘Approved’ status.

Going forward, we decided to apply these more stringent criteria across all the indexers of our articles to make it simpler for everyone.  What this does mean though is that there are a small number of articles published in 2012 that passed the old criteria but not the new criteria. It would be unreasonable to the authors to un-index them but of course those articles will not be indexed in PubMed as it stands, and most of those authors are working on new revisions as we speak.

Essentially, the editorial acceptance standard of F1000 Research was reset for the sake of indexing — not improved in a general way to match a standard peer review benchmark ala MEDLINE inclusion, but specifically dictated by PMC. So-called “open peer review” seems to have opened F1000 Research up to a detailed revision of its criteria.

Thanks to all of this, at F1000 Research, you can have an article that was published, indexed in all the places the journal was indexed at one point, and is now indexed in a subset of the places the journal is indexed once standards for acceptance changed.

More interestingly, PMC generated the standard of acceptance at F1000 Research. Instead of an indexing service being downstream from the journal when it comes to acceptance, PMC moved upstream. PMC’s specific standards became F1000 Research’s specific standards. That’s how upside-down things are at F1000 Research.

It took me a while to appreciate that F1000 Research has essentially outsourced its editorial authority — a major indexer has caused it to restate its standards of acceptance, making the indexer the new upper measure of validation. The effect shows itself in a new design on the F1000 Research articles landing page, which used to be just a running list, but which now sports a tabbed design with an “Indexed” tab and an “All Articles” tab. They might as well put “Accepted” and “Accepted and Rejected” on those tabs — the meaning would not change, but might become uncomfortably clear.

As I mentioned above, F1000 Research articles are apparently never set in stone, but elements like “published” and “indexed” and reviews are. But while they may not be set in stone by the publisher, they can suffer from neglect and indifference — something no paper in a pre-publication review system would suffer. Papers in traditional peer review systems either get an acceptance or rejection. There is no purgatory for papers to wander for eternity. But there is a purgatory at F1000 Research.

In my January 2013 post on F1000 Research, I noted that an article from November 2012 had not received any reviews. Checking back in April, the article still has not received any reviews. It is listed in the “All Articles” (or, “Accepted and Rejected”) tab. At what point (after six months?) will it be considered rejected? Or is it published but never accepted if it’s never reviewed and indexed (accepted)? In an email from November 2012, Lawrence responds to a question from PMC about the time period available for review in a way that can’t be reassuring to authors in this situation:

There is no end date to the process – that is one of the real benefits. Having said that, I expect it will be very rare that we will receive referee comments after a relatively short period following publication.

I would also take issue with her assertion that it’s a benefit to publish a paper that can be revised endlessly. The potential pitfalls with this were discussed above.

It’s also worth noting — mostly in case we ever pass this way again — that here is a government organization via PMC dictating a specific new standard around editorial acceptance for an applicant journal. At first, I hesitated to write this out, but the more I thought about it, the more odd it seemed. I couldn’t shake the feeling that this might be one of those “you saw it here first” moments. Instead of letting the thought pass, I guess it’s better to jot a note to the future that here we saw an OA publisher accept a new editorial standard from the US government.

F1000 Research seems to like to feel subversive. However, they have ceded their editorial authority to an index run by the US government. That’s not very subversive at all.

More importantly, the painful distortions to publication, peer review, and authority being perpetrated by F1000 Research makes pretzel logic seem straightforward. It’s no wonder that despite approval for indexing in PMC, no F1000 Research articles have appeared yet. Tagging this batch of illogic into a coherent system is a chore that NCBI and NLM technologists are finding quite perplexing. Even if they figure it out, whether published papers are ever reviewed, reviews pertain to the proper version, or indexed articles retain their indexed pedigree all remain major uncertainties.

I have no punchy ending to this post. F1000 Research just makes my head hurt.

Enhanced by Zemanta

About Kent Anderson

I am the CEO/Publisher of the STRIATUS/JBJS, Inc., the home of the Journal of Bone & Joint Surgery, JBJS Case Connector, JBJS Reviews, JBJS Essential Surgical Techniques, the JBJS Recertification Course, PRE-val, and SocialCite. Prior to this, I was an executive at the New England Journal of Medicine. I also was Director of Medical Journals at the American Academy of Pediatrics.

Discussion

21 thoughts on “Seeking Acceptance at F1000 Research — Early Problems With Identity and Outsourced Authority

  1. Would it make more sense to the F1000 Research process if all submissions started their life called a “pre-print?” Those that made it through the peer review process with sufficient certification would then be called “published.” I realize that this does little but change the terminology (since their papers are, for all intents and purposes “published”), but it does resolve the conundrum of submitted papers that sit in their system, either waiting for review or in a perpetual cycle of revision.

    At its heart, however, you do get into some of the fundamental challenges of post-publication peer review and whether F1000 Research can ultimately make it work.

    Posted by Phil Davis | Apr 25, 2013, 6:41 am
    • That’s one change that could help make F1000 Research less surreal. But I think the pre-print portion would need to be rebranded to avoid confusion in the market. Another positive change would be to put a hard stop to revisions to a paper. Another would be to improve their review and publication criteria. To me, F1000 Research demonstrates how sensible pre-publication review is.

      Posted by Kent Anderson | Apr 25, 2013, 6:55 am
    • You’d have to set up some kind of system for withdrawal of an article, or at least removing it from the review process. If I submit my paper as a “preprint” and it doesn’t get reviewed, and I then resubmit it to another journal, what happens if it the F1000 version subsequently gets reviewed and accepted? Do I get in trouble for publishing the same work twice?

      I would also worry about the impact of that label on the business model–will authors be willing to pay a fee to post a “preprint”? Does that label make it sound less definite that the paper will ever see review and acceptance?

      Posted by David Crotty | Apr 25, 2013, 7:00 am
      • Also I suspect some authors might seek a refund if there is a clear delineation offered. If I’ve paid for publication and my paper has failed to be published, rather than being in a nebulous pending state, I might want my money back.

        Posted by David Crotty | Apr 25, 2013, 7:34 am
      • With regards to business model, PeerJ Preprints notes that authors can “Submit a PrePrint for free” however they are built on a membership model so “free” in their case, means “included in your membership.”

        Posted by Phil Davis | Apr 25, 2013, 8:12 am
        • Phil, That is not correct. You get one preprint a year with a free membership account. You can sign up and create an account for free. If you publish you have to pay. You can even submit with a free account and join if it is accepted.

          https://peerj.com/pricing/

          Posted by David Solomon | Apr 25, 2013, 4:32 pm
      • The European Geosciences Union has a model like this. You submit to a “Discussions” version of the journal and when the paper is accepted, it is “promoted” to the Journal. But waiting for reviews is a critical problem and authors sometimes want to move their manuscripts to another journal. As a US publisher, we rejected those papers as previously published, regardless of whether they were going to be taken down from the EGU site. I think this gets to Bill Cohen’s point below – what is the definition of “publish”? I think you look to the root of the word – to make public. In that case, any manuscript posted on either the F1000 or the EGU site has to be considered published. Another publisher may make an exception to their rules on prior or dual publication, but that has to be done consciously.

        Posted by Bill Cook | Apr 25, 2013, 10:24 am
        • Instead of using the word-phrases “published” on the web can we say instead:

          “posted on the web”
          “mounted on the web”
          “deposited on the web”

          Or more twisted:

          “made accessible on the web”
          “made discoverable on the web”

          Or possibly insulting:

          “dumped on the web”

          Bill

          Posted by Bill Cohen | Apr 25, 2013, 10:36 am
          • We can call it anything we like, but it doesn’t change the basic idea: it was made available to the public and is therefore published.

            Posted by Bill Cook | Apr 25, 2013, 11:01 am
            • I liked “made available” even better, or possibly, “made accessible.”
              P.S. it may be too old-fashioned, but I still cling to the notion that if someone publishes something, one is also actively marketing, promoting or publicizing it. Too out-dated?

              Posted by Bill Cohen | Apr 25, 2013, 11:16 am
  2. Is “publish,” still, the right verb/word to use in this model? The papers sound more akin to poster presentations, or “exhibits” with little scratchpads where visitors can write commentaries.

    Posted by Bill Cohen | Apr 25, 2013, 8:40 am
  3. What is the point of these mega-journals? It seems to me they are a throwback to the world of society journals where members could publish pretty much any old thing so long as they paid the page charges. Except instead of page charges now we have whizzy new terms like APCs. If the society journals of 40 or so years ago had been so good, Robert Maxwell would have had more trouble blowing them out of the water than he did, wouldn’t he? At least society journals in those days made a pretence of peer review. And why is post publication peer review being touted as ‘new’?. Isn’t that effectively what happens anyway, when A tries to replicate B’s work? How is dispensing with pre-publication peer review helpful? Is the argument simply that it is plain unnecessary, and all those involved in it for all those years have just been wasting their time, the folk at F1000 know better?

    Posted by multisciencepublishing | Apr 25, 2013, 9:06 am
  4. “There are a lot of problems with the F1000 Research model.”

    “I have no punchy ending to this post. F1000 Research just makes my head hurt.”

    Those sentences in an excellent post pretty much sums it up.

    Posted by Adam Etkin (@adametkin) | Apr 25, 2013, 9:17 am
  5. So, under F1000’s criteria, it sounds as though an early paper about cold fusion might easily have been accepted in the first rush of enthusiasm, even though later on it might have received a long string of negative “not approved” reviews. I’m wondering, then, how F1000 really manages to distinguish science from non-science?

    Posted by Sandy Thatcher | Apr 25, 2013, 9:43 am
  6. How can one ever reliably cite an article if the content might be revised and altered at any time? How can you predict that the revised content will still meet your agreement or will still coincide with the point you’re making for citing it? Is the authors’ possibility to change their F1000 Research paper shut after it’s been officially accepted and indexed?

    Posted by Rachel | Apr 25, 2013, 10:10 am
    • Just to clarify, each version of an article is individually citable, which means that when you cite a paper, you are citing a specific version of that paper which in itself cannot change. Hence, if an older version of an article is cited, the reader is taken back to that version (with a note that comes up informing the reader that there is a newer version available).

      Posted by F1000Research (@F1000Research) | May 3, 2013, 7:28 am
      • If an article is revised, is it re-indexed and re-deposited in PubMed Central?

        Posted by David Crotty | May 3, 2013, 11:17 am
        • Yes; all versions subsequent to when an article is indexed (together with their associated referee reports and any author-referee discussion) will be deposited with PubMed Central.

          Posted by F1000Research (@F1000Research) | May 6, 2013, 4:07 pm
          • Thanks, good to know.

            Is there a set of criteria that must be met for redeposit? If I go in and change a punctuation mark in my paper, is it redeposited?

            How does PMC handle multiple versions of the same paper, presumably with the same title and same set of authors?

            Posted by David Crotty | May 6, 2013, 4:19 pm
  7. What’s happening at PMC is nothing that shouldn’t be expected. Every government compilation activity will have a problem with selectivity. Politicians don’t like to say “no!” Faced with pressure from someone excluded, ways will be found to weaken the requirements. The loss of selectivity is diffuse and much harder to quantify. It will not count as highly in the politicians’ minds.

    In contrast, one of a journal editor’s most important responsibilities is to enforce selectivity. Readers and the publisher of my journal expect only articles meeting certain theme and quality standards. They don’t specifically want me to make rejected authors unhappy, but that is a necessary consequence of selectivity. The managers of PMC and I just answer to different constituencies.

    Posted by Ken Lanfear | Apr 26, 2013, 9:46 am

Trackbacks/Pingbacks

  1. Pingback: Redundant and Expensive – How F1000 Research’s Model Reveals the Root Problems of PubMed Central | The Scholarly Kitchen - May 13, 2013

Side Dishes by Stewart Wills

Find Posts by Category

Find Posts by Date

April 2013
S M T W T F S
« Mar   May »
 123456
78910111213
14151617181920
21222324252627
282930  

The Scholarly Kitchen on Twitter

SSP_LOGO
The mission of the Society for Scholarly Publishing (SSP) is "[t]o advance scholarly publishing and communication, and the professional development of its members through education, collaboration, and networking." SSP established The Scholarly Kitchen blog in February 2008 to keep SSP members and interested parties aware of new developments in publishing.
......................................
The Scholarly Kitchen is a moderated and independent blog. Opinions on The Scholarly Kitchen are those of the authors. They are not necessarily those held by the Society for Scholarly Publishing nor by their respective employers.
Follow

Get every new post delivered to your Inbox.

Join 13,131 other followers

%d bloggers like this: