On Monday, the Faculty of 1000 announced a new service, the F1000 Journal Factor, that ranks the quality of journals based upon the ratings of individual articles submitted by volunteer faculty reviewers.
While still in beta, the new service is a clear shot across the bow of Thomson Reuter’s journal impact factor.
Unlike the impact factor — released by report each summer for the previous year — the F1000 Journal Factor is updated monthly. It is designed to “provide a continuously updating picture of journals ranked by excellence within biology and medicine,” according to the F1000 press release.
In comparison to the impact factor, which simply reports the average number of times an article has been cited over a two-year window, the F1000 Journal Factor is much more elaborate. It’s a calculation that combines individual article rankings given by faculty reviewers with the frequency of reviews in their system.
The Journal Factor requires a few paragraphs to describe, but if we’re to understand what it measures, it’s important to deconstruct the metric. In its simplest form:
Journal Factor = log10 ((Sum of Article Factors) x (Normalization Factor) +1 ) x 10
An Article Factor is calculated by taking the highest article rating (10=exceptional, 8=must read, 6=recommended) to which an incremental value is added for each additional rating (3, 2, or 1, respectively). So if an article received one exceptional rating and two recommended ratings, its score would be 10+1+1=12. All of these article factors are then summed up for each journal.
The Normalization Factor is the proportion of articles in a journal that received at least one review. Like the calculation of the impact factor, it is designed to provide equal weighting to journals of different sizes. As some journals will have received zero ratings over the period of evaluation, they add 1 to all journals so that their logarithmic transformation (explained next) would not result in error.
The logarithmic transformation (log10) takes a distribution that extends over several orders of magnitude and draws that long-tail in. (As an analogy, earthquake data is based on log10, meaning that a 9 degree quake is 10 times more powerful than a 8 degree quake and 1000 times more powerful than a 6 degree quake). All final scores are then multiplied by 10, as the company describes, “to make the FFj a readable number.”
There are a few things that concern me about their calculation.
First, the calculation gives much greater influence to articles receiving just one rating (and particularly a high one) than those articles that receive multiple ratings. The result is a Journal Factor that is highly sensitive to enthusiastic reviewers who rate a lot of articles in small journals.This may explain why the Journal of Sex and Marital Therapy  ranks higher in F1000 rankings than Science, PNAS, or Cell.
Second, I’m a little uncomfortable with what the logarithmic transformation does with the interpretation of the journal ratings. Even after applying a journal size normalization factor, the distribution of journal scores is highly skewed, with a few journals receiving the vast majority of article evaluations and a very long tail of infrequently evaluated (or unevaluated) journals. The log transformation obscures that distance.
Third, the F1000 Journal Factor attempts to rank journals based on a proportionally small dataset of reviews by a proportionately small group of reviewers:
On average, 1500 new evaluations are published each month; this corresponds to approximately 2% of all published articles in the biological and medical sciences.
This probably explains why the company gives a lot of weight to journals with just one article review and finds it necessary to perform a logarithmic transformation to the data.
To be fair, the F1000 business model was not designed to provide comprehensive reviews, so a lack of completeness should not be considered a failure on their part. The purpose of F1000 is to guide readers, through expert evaluation, to a small sub-population of worthwhile literature. To derive a comprehensive journal ranking system built on such limited data, however, is to build a map where 98% of the territory remains unexplored. Where ancient cartographers would have filled in these empty sections of the map with pictures of savages and sea-monsters — so that the reader understands what remains uncharted — these Journal Factors assume that missing data are the same as zero. They are not the same.
From a business standpoint, I’m not surprised at this new development. F1000 is attempting to produce a new derivative product from an existing service, which is no different from Thomson Reuters creating derivative products, like the Eigenfactor or 5-year impact factor, from their own citation data.
From a rhetorical standpoint, however, F1000 is taking a 180-degree turn. From its inception, F1000 was publicly dismissive of the value of the journal. The purpose of F1000 was to direct readers to important articles irrespective of the imprimatur. This was — and still is — the principal argument behind post-publication peer review. What is curious is that the motto of F1000 is now gone from the company’s website, although you will still find it verbatim on Facebook, Wikipedia and some library websites.
F1000 rates research articles on their own merits rather than according to the prestige of the journal in which they are published (the impact factor)
If F1000 were able to adequately deal with issues of completeness and potential conflicts of interest, their Journal Factor may ultimately not tell us anything that we don’t know from Thomson Reuter’s impact factor, which is, if you want to read good articles, you will find most of them in good journals.
In sum, the real value of F1000 is not what the aggregate data can tell us about individual journals, but in what experts can tell us about individual articles. As local guides, faculty reviewers know much more about the territory than does the cartographer.
 R Taylor Segraves was responsible for submitting 23 of the 34 article reviews for the Journal of Sex and Marital Therapy. He is also the editor-in-chief of that journal.