Aug 18, 2010

Reference List Length and Citations: A Spurious Relationship

Posted by Philip Davis under Education, Metrics and Analytics, Research, Sociology | Tags: , , , , , , |
[10] Comments 

The world is full of strange associations.  For instance, ice cream sales are highly correlated with the U.S. murder rate, but no one in their right mind would suggest that ice cream is responsible for violent behavior — that would give real meaning to “death by chocolate.”  The underlying cause that connects murder with ice cream sales is heat.  Heat makes people irritable and irrational, and given the right conditions, may lead to violent behavior.   There is no theoretical basis of ice cream leading to violent behavior — this relationship is merely spurious.

The length of a paper’s reference list is also associated with the number of citations that a paper receives, writes Zoe Corbyn in this week’s Nature News (“An easy way to boost citations,” Aug 13), reporting on a study of more than 50,000 papers published in the journal Science.  This association, writes Corbyn, provides evidence of widespread corruption in the citation process by involving a mutual “I’ll cite your paper if you cite mine” tactic designed to benefit both the citing and cited authors. She writes:

“There is a ridiculously strong relationship between the number of citations a paper receives and its number of references,” Gregory Webster, the psychologist at the University of Florida in Gainesville who conducted the research, told Nature. “If you want to get more cited, the answer could be to cite more people.”

Webster’s findings are based on statistical correlation — a simple measure of the association between two variables — and his plots look pretty impressive. But there is something odd about this relationship.  I’ve never consciously cited papers because they include long reference lists.  In fact, if you ask me about some of my favorite papers — the ones I cite over and over again — I’d be at a loss to give you even a ballpark estimate of their reference length.  Number of references never comes into my mind as a rationale  for citing an article.  So why the correlation?

In the next few paragraphs, and with the help of my own dataset of Science articles published in 2007, I will demonstrate that this relationship between references and citations is entirely spurious.  First I’ll replicate the association.  Then I’ll make it go backwards.  And for a final trick, I will make the association disappear completely.  (If you are impressed, you can hire me for your kid’s next birthday party).

Citations vs. References in Science Magazine (2007

Ladies and gentlemen, on your right is my scatter-plot — similar to Webster’s (p.10) –  showing the positive relationship between references and citations.  Even for articles aged two years, the relationship is statistically significant (p<0.001), although it can explain only 4% of the total variation in article citations.

Now, let’s add to this simple, predictive model another variable: length of article in pages, since longer papers tend to cite more references.

As you’ll see in my second plot (below), the act of adding the length of the article changes the direction of the relationship.  The relationship between references and citations is now negative.  More references now lead to fewer citations, and this relationship is highly significant (p=0.0077).

By simply adding the number of pages to the model, I have turned a significantly positive relationship into a significantly negative one, and in doing so, I can also explain another 12% of the variation in article citations.

References vs. Citations controlling for Page length

For my last trick, I’m going to make the relationship between reference length and citations completely disappear.  To do this, I need to build a more realistic model.  I’m going to add two more variables to the statistical model: the number of authors, and the subject section of the article.

Science is a multi-disciplinary journal, publishing articles in the biomedical sciences, physical sciences, and, to a lesser degree, the social sciences. By including the subject of the article, we can control for the differences in citation rates across the disciplines.  The section headings come straight from Science magazine — there is no easy field that can be used from ISI’s Web of Science.   Below are my abbreviated model results (Note: Ln stands for the natural logarithm of each variable).

Full model summary

Controlling now for pages, authors, and section, reference list length is no longer statistically significant.  In fact, it looks pretty much like a random variable (p=0.77) and adds no information to the rest of the regression model.  Page count and number of authors are still highly significant.  What you’ll note from the sum of squares is that the article’s section is, by far, the strongest predictor of article citations in our model. Adding section to our reference model allows us to explain a full 46% of citation variance.

So there you have it. The simple association between reference length and citations is spurious, confounded by such other simple explanations such as the length of the paper and the section in which it was published.  There may still be evidence of mutual citation gaming in science, although this study cannot  validate these claims.

Enhanced by Zemanta
 

Aug 16, 2010

Ending the Supplemental Data “Arms Race”

Posted by Philip Davis under Authority, Controversial Topics, Experimentation, Peer Review | Tags: , , , , , |
[18] Comments 
Data Dump
DATA DUMP by swanksalot via Flickr

In science, data is good.  So more data should be better?

Not according to the Society for Neuroscience, which has decided to stop accepting and hosting supplemental data with journal articles.

Announcing their decision in the August 11th issue of the Journal of Neuroscience, Editor-in-Chief John Maunsell explains that the rationale was not about space and cost, but scientific integrity.  Supplemental materials have begun to undermine the peer review process.

Validating supplementary data adds to the already overburdened job of the reviewer, Maunsell writes.  Consequently, these materials do not receive the same degree of rigorous review, if any at all.  At the same time, the journal certifies that they have been peer-reviewed.

Since 2003, when the Journal of Neuroscience began accepting supplemental data, the average size of these files has grown exponentially and is rapidly approaching the size of the articles themselves.

With few restrictions on space, reviewers may place additional demands on authors, requiring them to perform and add new analyses and experiments to the supplemental data. Often these additions are “invariably subordinate or tangential,” Maunsell maintains, but represent significant work from the author and thus delay the publication process.  Supplemental data thus changes the expectations of both author and reviewer, leading to what he describes as an “arms race:”

Reviewer demands in turn have encouraged authors to respond in a supplemental material arms race. Many authors feel that reviewers have become so demanding they cannot afford to pass up the opportunity to insert any supplemental material that might help immunize them against reviewers’ concerns.

In the August 11th issue of the Journal of Neuroscience, 19 of the 28 original articles contained supplemental materials, suggesting that they have become normal parts of the publication process.

Yet, having two places to report methods, analyses, and results compromises the narrative of the article as a “self-contained research report,” Maunsell argues.  Instead of a clear scientific story (e.g. this is the problem, this how we approached it, this is what we found and why it is important), the article becomes a kitchen sink, a collection of related, but often unimportant details, attached to a clear story.

As Maunsell explains, there are no viable alternatives to simply ending the practice of accepting supplemental materials.  Limits on the number of additional tables and figures are simply arbitrary; stating that only “important” additions be included makes enforcement impractical. The journal sees no alternative to ending the practice entirely.  This doesn’t mean that authors cannot host their own supplementary data, with links from the article — the journal simply will not vouch for their validity or persistence. Rationalizing the decision, Maunsell writes:

We should remember that neuroscience thrived for generations without any online supplemental material. Appendices were rare, and used only in situations that seemed well justified, such as presentation of a long derivation.

The decision to end a seven-year practice of accepting supplemental scientific data is fascinating if viewed within the larger framework of science policy that has been dominated lately by frameworks of openness and transparency.

Recent developments such as Open Notebook Science, Science Commons, Open Source, Open Access, and Open Government preside under the notion that more access to data allows for greater efficiency and greater accountability.  What it ignores, however, is that access is not enough.

Readers seek filters that attest to a source’s validity (the backlash over WikiLeaks’s publication of thousands of unverified documents may signal that the established news organizations are still valued for their ability to verify the authenticity of facts and stories). The decision of a journal to cease publishing supplementary materials affirms the same position that all the facts of science should pass through the same filter.

Enhanced by Zemanta
 

Aug 10, 2010

The Mismeasure of Man, Funds, and Open Access Experiments

Posted by Philip Davis under Controversial Topics, Education, Experimentation, Metrics and Analytics | Tags: , , , , , |
[8] Comments 
First edition (1981) of The Mismeasure of Man

Image via Wikipedia

How much does open access publishing cost an institution?  “Almost nothing,” according to computer scientist and open access advocate, Stuart Shieber.  The reason?  Most open access publishing funds have sat unused.

Writing in his blog, The Occasional Pamphlet, Shieber provides the number of open access articles paid for by institutions participating in the Compact for Open Access Publishing Equity (or COPE), and the numbers are dismally low.  Since inception, nearly one year ago, Columbia and Cornell have sponsored two and three articles, respectively.  Dartmouth and Harvard have sponsored just one article, and MIT and the Memorial Sloan-Kettering Cancer Center both stand at zero.

Two institutions are exceptions to this trend — Berkeley (92) and Ottawa (25) — both of which will cover articles published in hybrid journals (subscription journals that permit authors to make their articles freely accessible).  The other COPE signatory institutions restrict their funds to full open access journals.

The results of these findings are not that surprising.  Most faculty do not consider open access publishing high on their academic priority list (see the Berkeley report and the Ithaka report), and those who do often have access to their own research funds to support this type of publishing.  What is surprising is how Shieber spins the results:  Limited use is a success –  not a failure — and the venture is cheap if viewed from a cost-per-faculty vantage point.  As he concludes:

The bottom line is that the direct costs of running a COPE-compliant open-access fund are trivial, and the administrative costs of dealing with handfuls of requests are trivial as well.

There is something very odd about this way of thinking. Public service programs are often considered failures when they sit on large sums of unspent money. Unused funds signal a conspicuous lack of demand, a sign that a service wasn’t required and that the money could have been better spent another way. Most funders require unspent funds to be returned, and this act often justifies a budget reduction in the next fiscal year.

In the case of Cornell’s Open Access Publication Fund, $50,000 could have been used to avert the cancellation of journals and databases, could have purchased hundreds of books, or could have been used to support a fledgling service where there was a strong indication of demand.  This $50,000 was money diverted from the funds of selectors to purchase library materials and services, then left to sit, unused, in a special account where it benefited no one.

Yet using a very strange form of accounting, Sheiber argues that these few publication expenditures  represents peanuts when calculated as a cost-per-faculty basis.  Cornell, for example, spent only $3.08 per faculty member and Harvard only $1.  Sounds like a good deal, right?

Sheiber comes up with these figures by multiplying the number of sponsored articles by $1,500 (his estimate for the author-side publication costs), adjusts for the amount of time that the program has been in place to come up with an annual figure, and then divides the sum by the number of university faculty to come up with an annual cost per faculty number.

What’s wrong with this measurement? Lots.

First, the real cost to Cornell for its open access publication fund is $50,000.   If only three faculty used this fund, the cost of this fund per faculty was $50,000/3 or $16,667 per faculty member.  It makes little sense to divide total costs into the entire faculty since the entire faculty were not beneficiaries of this fund.  Similarly, the $45,500 left in the Cornell COPE account should not be ignored.  As I mentioned, these were funds that could have gone to purchase materials and services for the Cornell community.  This fund represents unrealized capital, and by hording it, it created harm (not benefit) to the Cornell faculty.

The Compact for Open Access Publishing Equity has been promoted as an “experiment” in scholarly communication, and yet this experiment appears to be immune from any form of reasonable evaluation.  For instance, the data show strong support for the hybrid-OA publication program offered by Berkeley and Ottawa, yet Shieber is unwilling to cede an inch to a transitional hybrid model, calling it “double-dipping.”

There is much to learn from experiments such as COPE.  We should treat them as learning experiences that can help us guide future policy, and not as pet projects that require twists of logic and poor accounting to justify their existence.

Enhanced by Zemanta
 

Aug 6, 2010

Pubget Continues to Puzzle and Diversify

Posted by Philip Davis under Business Models, Commerce, Technology, Usability | Tags: , , , , |
[2] Comments 

I’ve never understood the rationale behind Pubget.

I accept the premise that more scientific articles are published each year and that scientists are far too busy to read even a fraction of them. What I just don’t understand is how bulk downloading of article PDFs is a solution to the problem of overabundance.  The time spent browsing and evaluating the merit of an article far outstrips the time it takes to download the file.

In last year’s coverage of Pubget, I questioned whether their business model, “based on selling advertisements to lab equipment and pharmaceutical companies,” was viable.  I also questioned whether this model was parasitic on the very same companies that produced the content in the first place and whether publishers would participate in such a venture.  I was doubtful that this company would last the year.

Yet they are still in business, still growing, and have diversified their services.  They are no longer on a beta test site, but a fully-operating service.  It’s clear that I understand little about the information industry.

According to Ramy Arnaout over 200 libraries have registered their IP ranges with Pubget, permitting access to subscription-only articles through their service.  And last summer, Pubget partnered with Seed Media to provide free articles to science bloggers at ResearchBlogging.org

Like many online ventures banking on revenue from advertising, Pubget has been unable to attract many paying customers.  Many of the ads in Pubget point to other Pubget services.  Perhaps the drought of advertising dollars has pushed Pubget into rolling out new services (Pubget Premium Services) geared at bringing in other sources of revenue.  One such service, PaperStats, functions to help librarians aggregate and analyze publisher usage data.  (It’s somewhat ironic that the company that has built its search and delivery system around a bulk downloading application has gotten into the cost-per-use game.)

Price markup

Pubget has also attempted to generate service fees by selling journal PDFs to non-subscribers.  The price markup of these articles is significant.  For example:

  • Cell, Current Biology and The Lancet ($54.50 from pubget; $31.50 from the publisher)
  • JAMA ($35 from Pubget; $30 for 24hr access from the publisher)
  • Nature ($47 from Pubget; $32 from the publisher)
  • New England Journal of Medicine ($33 from Pubget; $15 from the publisher)
  • Science ($35 from Pubget; $15 from the publisher)

Arnaout explained that the difference in pricing is partly based on copyright clearance and partly on what the publisher sets as royalties.  Unlike DeepDyve, there is no way to preview an article before purchase. A Pubget user has little more to go on beyond the abstract, if available.

Selling free articles

While Pubget doesn’t attempt to sell content from open access journals (articles from PLoS journals are still free), it does attempt to sell freely accessible articles in subscription-based journals.

For example, the August 4th issue of JAMA leads with two articles (an editorial and an original paper), both of which are free from the website but require purchasing ($35 each) from Pubget.   Similarly, a free editorial in the August issue of The FASEB Journal goes for $28 on Pubget.  Many of the author-pays open access articles in PNAS are free in Pubget, but not all.  And, the one free “Featured Article” in the July 27th issue of Current Biology costs $54.50.

Paying for free content: JAMA vs. pubget (Click for details)

I asked Arnaout about these inconsistencies.  He explained that Pubget is transitioning to a new system and to expect a few “hiccups” from time to time.  They have no intention to charge for free articles.

Advanced search

What is also new to Pubget since my last review is an advanced search engine that allows — in theory — more refined searching than a relevance-based keyword search on the article’s metadata.  I say “in theory” because I couldn’t get it to work.  Specific journal searches yielded zero results.  Limiting one’s search by volume, page number, or year limits also failed to work.   Unlike PubMed (the free source of much of Pubget’s metadata), there was no way to build a search or conduct a post-search revision.  It might have been more useful to start with PubMed and then add some functionality, which is also what Pubget has done with PaperPlane.

PaperPlane

Pubget has developed a browser plug-in called PaperPlane, which adds the core function of Pubget (bulk downloading of PDFs) directly onto PubMed.  Personally, I find this solution much more appealing since it adds functionality to a robust and fully-functioning resource.  PubMed, however, links directly to the publisher’s website to access the full-text (HTML) article, which is the version that provides the most functionality for readers.  The price of using PaperPlane is giving up on the extended functionality of the fulltext article for the flat, archival PDF version.

In sum, Pubget still seems like a solution based on a simple, old technology of file transfer (the FTP “Get” command) in desperate search of a problem, or a problem that doesn’t exist.  The diversification of this company over the last year may signal a fledgling Internet startup company still in search of a viable business model.

Enhanced by Zemanta
 

Jul 22, 2010

No Deposit, No Diploma: How Graduate Schools and Libraries Restrict Access to Dissertations and Theses

Posted by Philip Davis under Authority, Social Role, Technology | Tags: , , , , , , |
[15] Comments 
its flaming hoop time

Flaming hoop by Terwilliger911 via Flickr

The last thing a graduating student wants to do before receiving his diploma is fight the system.  After all, to paraphrase Mike Myers as Dr. Evil, “I didn’t spend six years in graduate school to be called ‘Mister,’ thank you very much.”

And yet the classes, teaching, research, writing, and exams were pretty tame compared to the last flaming hoops I was required to leap through in order to receive that piece of paper with my name followed by PhD.

These flaming hoops had to do with filing my dissertation — a document I’m convinced few will read with the exception of my advisory committee and perhaps my father as a cure for insomnia.

As I tried to explain to my father, who has requested several times for a document draft, dissertations are not meant to be read.  They contain all of the data, in nauseating detail, a graduate student generates in pursuit of a research question, including failed experiments, supplementary analyses, data tables, and an exhaustive literature search that is designed to exhaust the reader, or at least provide some evidence that all relevant sources — however tangential — have been consulted.

In sum, dissertations are everything that a doctoral student’s professors want to see, although this does not translate into what a reader wants to read.  The reader wants a couple of thousands of words, a table or two, a figure, and a nice short summary.  In other words, readers want a journal article, which is exactly what I plan on spending my time writing as a postdoc.  But before becoming a postdoc, I first need to become a “doc.”

And the last flaming hoop that I am required to jump through is . . . filing my dissertation.

This is where the story gets interesting.  But please allow me one more short digression.

The dissertation (often called a thesis) is an old tradition that still persists in many academic fields, although many departments have replaced it with a papers option, something more contemporary with modern scientific publishing.  Dissertations are public documents, and within a few weeks, two hardbound copies (printed on archival paper with gold print on their handsome black spines) will show up on our library shelves.  You are welcome to read them, but you first need to get to Ithaca, NY.  Alternatively, ProQuest-UMI will print and sell you a copy.  But don’t bother, because a publicly-accessible file will show up in eCommons, our institutional repository, or at least this is what I’m trying to make happen. Yet, the system is working against me, providing all the wrong incentives that push me to embargo access to this file for a very long time.

On submitting my dissertation PDF, I have just two options: embargo access to the file for 5 years (the default), or make it available immediately.

The student daring enough to change the setting to “open circulation” is then presented with a warning page with an option to go back and return the settings to the default.

It shouldn’t be surprising that the vast majority of graduate students have chosen the default.  Only 11% (16 of 133) dissertations filed this year made their document immediately available, and I’ll become the 17th.  While theses and dissertations belong to the largest and fastest growing collections in our institutional repository, it shouldn’t surprise you that most of these files aren’t accessible at all.

Did I tell you that Ithaca is very beautiful at this time of the year?

The ProQuest-UMI submission was a more challenging hoop.  If I wanted them to make my dissertation open access, it would cost me US $160 (students have to pay UMI to index their theses and dissertations).  Our Thesis Advisor even scratched out the open access checkbox on my form with a red pen and only relented to accept my decision after I convinced her that I understood the consequences.  Without looking up, she handed me a payment slip and sent me off to the Bursar’s Office to pay the fee.

On the way to the Bursar’s Office, I tried to contemplate how the framers of our access policy settled on five years and why students are required to opt-out rather than opt-in to the embargo.

In my search for the answers, I discovered that many of the people  involved in making this policy have either left Cornell or died.  Those who remain, like the Thesis Advisor, simply continue the tradition, citing a warning that public access will dash any hope of future publication.  A graduate student with any hope — even remote — of publishing after publication of his or her thesis should agree to embargo.

And yet, I could not.

I concluded that after spending a year on writing a public document, only to make it as inaccessible as possible, was antithetical to the purpose it was intended to fulfill.   Indeed, consigning to lock up a dissertation on open access seemed inconsistent, if not downright hypocritical.  And lastly, I have yet to find a publisher who has a hard policy against accepting a manuscript that was derived from a graduate thesis.  While most serious journals have statements about only accepting original material, filing a dissertation does not seem to constitute “prior publication,” or at least none are willing to state this.

Information policy is no different from the Strange Laws of Old England that persist even when technology and social mores have changed.   If universities and libraries are serious about making research publicly available, they should consider revising old policies to actively promote this goal.

Because graduate students are willing to jump through any hoop.

Enhanced by Zemanta
 

Jul 14, 2010

Post-Publication Review: Does It Add Anything New and Useful?

Posted by Philip Davis under Authority, Controversial Topics, Metrics and Analytics, Research | Tags: , , , , , |
[64] Comments 

Post-publication review is spotty, unreliable, and may suffer from cronyism, several studies reveal.

Reporting last month in the open access journal Ideas in Ecology and Evolution [1], ecologist David Wardle analyzed over 1,500 articles published in seven top ecology journals during 2005 and compared their citation performance nearly five years later with initial Faculty of 1000 (F1000) ratings.

Faculty of 1000 is a subscription service designed to identify and review important papers soon after they are published.  F1000 reviewers assign one of three ratings to journal articles: (“Recommended” = 3; “Must read” = 6; or “Exceptional” = 9) plus a brief comment.  For papers that receive more than one rating, a F1000 Factor is calculated.

Wardle reports that fewer than 7% of articles in his study (103 of 1,530) received any rating, with only 23 receiving a “must read” or “exceptional” rank.

Moreover, F1000 ratings predicted poorly the citation performance of individual articles.  The top 12 cited articles in his study did not receive any rating at all, and many of articles that received a high rating performed poorly.  Wardle concludes:

If, as this analysis suggests, the F1000 process is unable to identify those publications that subsequently have the greatest impact while highlighting many that do not, it cannot be reliably used as a means of post-publication quality evaluation at the individual, departmental, or institutional levels.

Speculating on why an expert rating system claimed to involve the “leading researchers” in science operated so poorly, Wardle offered several explanations.  First, coverage of the ecological literature in F1000 is spotty, with some subfields completely ignored.  Second, he believes there is evidence of cronyism in the system where F1000 section heads appoint their collaborators, colleagues and recent PhD graduates, many of whom share similar views on controversial topics.  The appointing of F1000 raters, Wardle adds, appears to suffer from geographical bias, with North American section heads appointing North American faculty who subsequently recommend North American articles.  He writes:

The F1000 system has no obvious checks in place against potential cronyism, and whenever cronyism rather than merit is involved in any evaluation procedure, perverse outcomes are inevitable.

In a paper published last year in PLoS ONE [2], five members of the Wellcome Trust analyzed the citation performance of nearly 700 research articles receiving Wellcome funding with F1000 scores.  Each paper was also rated by two different researchers at the Wellcome Trust for comparison.

Similar to Wardle, F1000 faculty rated just under 7% of the cohort of papers.  While there was moderate strength correlation (R=0.4) between Wellcome Trust ratings and F1000 ratings, many highly-rated articles identified in one group were completely dissimilar from the other group, and less than half of the most important papers identified by Wellcome raters received no score from F1000.  Allen et al. write:

. . . papers that were highly rated by expert reviewers were not always the most highly cited, and vice versa. Additionally, what was highly rated by one set of expert reviewers may not be so by another set; only three of the six ‘landmark’ papers identified by our expert reviewers are currently recommended on the F1000 databases.

An alternative to Impact Factor?

When articles receive multiple F1000 reviews, their average rating does not differ substantially from their journal’s impact factor, a 2005 study concludes [3].  An analysis of 2,500 neurobiology articles revealed a very strong correlation (R=o.93) between average F1000 rating and the journal’s impact factor.  Moreover, the vast majority of reviews were found in just 11 journals.

In other words, F1000 ratings did not add any new information — if you are seeking good articles, you will find them published in good journals.

While advocates of post-publication review may counter that this type of metric is still new and undergoing experimentation, F1000 Biology has been in operation since 2002, adding new faculty and sections ever since.  In 2006, the company launched F1000 Medicine.

Earlier this year, Lars Juhl Jensen, a computational biologist and author of several PLoS articles, analyzed the post-publication statistics made publicly by PLoS and reported that user ratings correlated poorly with every other metric — especially with citations — and wondered whether providing this feature was useful at all.

Unless post-publication review can offer something more expansive, reliable, and predictive, measuring the value of articles soon after publication may be more difficult and less helpful than initially conceived.

—-

[1] Wardle, D. A. 2010. Do ‘Faculty of 1000′ (F1000) ratings of ecological publications serve as reasonable predictors of their future impact? Ideas in Ecology and Evolution 3, http://dx.doi.org/10.4033/iee.2010.3.3.c

[2] Allen, L., Jones, C., Dolby, K., Lynn, D., & Walport, M. 2009. Looking for Landmarks: The Role of Expert Review and Bibliometric Analysis in Evaluating Scientific Publication Outputs. PLoS ONE 4: e5910, http://dx.doi.org/10.1371%2Fjournal.pone.0005910

[3] Editor, A. 2005. Revolutionizing peer review? Nature neuroscience 8: 397, http://dx.doi.org/doi:10.1038/nn0405-397

Enhanced by Zemanta
 

Jul 12, 2010

Impact Factor Inflation: When an Increase is Actually a Decrease

Posted by Philip Davis under Marketing, Metrics and Analytics, Research | Tags: , , , , |
[5] Comments 
Inflation
Inflation by darkmatter via Flickr

Every June, publishers and editors anxiously await the Parade of the Journal Citation Reports, and hope  — some even pray — that their own numbers will increase.

After all, an increase is an increase, right?

In spite of a plea from one liblicense-l reader, it didn’t take long for publishers to begin crunching 2009 performance figures to see how well they did.  And from several press releases, many publishers did very well, for example:

So when is an increase not always beneficial?

Writing in the June issue of BioScience [1], Bryan Neff and Julian Olden, two biologists, argue that journal impact factor increases must be viewed against general background inflation.   They write:

. . . much in the same way a modest salary raise effectively means very little in an increasingly costly economy, an increase in a journal’s impact factor must be interpreted with respect to background levels of inflation.

In examining 70 journals in ecology, they found that almost 50% showed increases in their impact factors, but at rates lower than the background inflation rate.  In other words, they were failing to keep pace with inflation.

Exploring the principle causes of impact factor inflation, Neff and Olden discovered that ecology papers are citing more papers — seven more, on average, in 2007 than a decade earlier.  More importantly, proportionately more of these citations are to recent papers — papers eligible to count towards a journal’s impact factor calculation.

Reporting last year on a much larger sample (the entire list of journals in ISI’s Journal Citation Reports, or 4,300 titles), Althouse et al. [2] reported that 80% of the journals listed in the JCR showed an impact factor increase between 1994 and 2005, growing by an average rate of 2.6% per annum.  Indeed, all subject disciplines revealed inflation with the exception of two: History, and History and Philosophy of Science, both of which showed negative growth.

Like Neff and Olden, Althouse reports that the chief explanation for general inflation is that reference lists have continued to get longer, although cross-disciplinary differences in impact factors could be partially explained by the proportion of references citing papers published within the last two years.

In 1969, Nature’s first impact factor was 2.343 and only 10 journals had an impact score of 10 or over [3].  Given the compound nature of inflation, it shouldn’t take too long to see triple-digit figures.

—-

[1] Bryan D. Neff and Julian D. Olden (2010). Not So Fast: Inflation in Impact Factors Contributes to Apparent Improvements in Journal Quality. BioScience 60(6):455-459. http://caliber.ucpress.net/doi/abs/10.1525/bio.2010.60.6.9

[2] Althouse, B. M., West, J. D., Bergstrom, C. T., & Bergstrom, T. 2009. Differences in impact factor across fields and over time. Journal of the American Society for Information Science and Technology 60: 27-34, http://dx.doi.org/10.1002/asi.20936

[3] Garfield, E. 1972. Citation analysis as a tool in journal evaluation. Science 178: 471-479, http://dx.doi.org/10.1126/science.178.4060.471

Enhanced by Zemanta
 

Jun 21, 2010

PLoS ONE: Is a High Impact Factor a Blessing or a Curse?

Posted by Philip Davis under Business Models, Controversial Topics, Metrics and Analytics, Peer Review, Research, Sociology | Tags: , , , , , |
[70] Comments 
PLoS ONE logo
Image via Wikipedia

Last week PLoS ONE received its first impact factor — a stunning 4.351.  This puts the open access journal in the top 25th percentile of ISI’s “Biology” category, a group of journals that sports a median impact factor of just 1.370.

Within minutes after the impact factors for 2009 being released, one could almost hear the sounds of champagne corks popping in San Francisco.  Or were they the sounds of competing publishers banging their heads against their office walls in utter amazement?  For there seems something very strange about a journal that accepts nearly 70% of all submissions yet achieves such a score, especially with its first assessment.

For anyone considering that PLoS ONE engaged in editorial shenanigans to boost self-citations, there is no evidence of such.  Self-cites represent just 8% of the citations used to calculate their impact factor, and removing these self-cites drops their score to just 4.00.  We must accept that this is a valid and correct measure of the citation impact of PLoS ONE articles.

So how can a journal that allows 7 out of 10 manuscripts through their gate achieve such a stellar rating?

We need to remember that it costs $1,350 to publish an article in PLoS ONE. And while their article processing charges (APCs) are not as high as competing commercial publishers’, they still exert a barrier to authors. Authors with access to funds (either through their own research grants or through their institution’s open access publishing funds) represent a characteristically different group of authors than the rest of the author population. (I’ll address APC waivers later in this post).

Authors with research funds have already gone through a form of peer-review by their granting agency, which has selected their proposal above all others because it has merit and stands a chance of producing interesting results.  And authors with access to publication funds through their institution have passed through the most finest filter of them all — the peer-review that rewarded them with employment at an elite institution in the first place.

Either way, being willing and able to pay $1,350 to pay to publish an article is a signal that there is something unique about PLoS ONE authors, and it shouldn’t be so surprising that articles written by the 70% who were able to pass through the publication review gate are performing so well.

Is a High Impact Factor a Blessing or a Curse?

Normally, publishers would view a high initial impact factor as a blessing.  After three hard years, scientists have recognized that PLoS ONE is a source of quality articles that are worth both their attention and citations.  More importantly, authors who published early with PLoS ONE did so for reasons other than an impact factor.

For many authors, a journal’s impact factor strongly determines where to submit one’s manuscript.  Authors initially skeptical that a journal working on the principle of light peer-review may now consider PLoS ONE a destination for future manuscripts. PLoS ONE should prepare for the deluge.

While a subscription-access publisher may dream of being overwhelmed by manuscript submissions, this blessing may come as a curse for a publisher that has banked on making money from not spending a lot of time and resources rejecting manuscripts.

In financial terms, the purpose of PLoS ONE is to subsidize the cost of publication for their two flagship journals, PLoS Biology and PLoS Medicine, two highly-selective, high-cost journals that would be unable to continue in their current state without financial subvention.  PLoS ONE is able to do this because of low selectivity — rejecting only 3 out of every 10 manuscripts.  In order to make their parent organization money, therefore, they need to rely heavily on automation and minimal intervention from the publisher.  For PLoS ONE, being a “bulk publishing” journal serves a clear purpose and one that should not be scorned.

Living with an Impact Factor

On the face of it, it would appear that the future of PLoS ONE is rosy: increased submissions leads to increased revenue and more support for flagship journals for which PLoS is chiefly interested in maintaining as highly-selective journals.

This future, however, is based on two assumptions:

  1. The quality of PLoS ONE submissions remains constant, and
  2. No more free riders

As mentioned above, the fact that PLoS ONE now has a respectable — even enviable — impact factor will draw a different kind of author who may not have considered this journal in the past.  If this different author is like the kind of authors that most conventional journals routinely deal with, PLoS ONE may need to be more discerning of which manuscripts are let through their gate; that is, if they care about their future impact factor.  If they do care, then a declining quality of manuscript submissions means that PLoS ONE will need to spend more time and energy reviewing manuscripts only to reject many of them without payment.  This will only drive up the cost of running their operation, or alternatively, dig into the surplus they send back to support the operations of PLoS Biology and PLoS Medicine.

Secondly, and more importantly for PLoS ONE, their financial model requires that the same proportion of authors will pay article processing fees.  But because PLoS has separated editorial decision-making from their business model, they have little sway in keeping payments flowing if authors decide they are either unwilling or unable to pay.  From their Author Guidelines page:

Editors and reviewers have no access to payment information, and hence inability to pay will not influence the decision to publish a paper. These policies ensure that the fee is never a barrier to publication.

How many PLoS ONE authors claim financial hardship?

While I cannot find public information on the percentage of waivers or discounts given to authors who express financial hardship, I was told by a PLoS publisher their system works exceptionally well, with only a small percentage asking for — and receiving without question — article processing waivers.

Now there might be a valid reason for not making waiver statistics public.  With a “no questions asked” policy, the system essentially works on voluntary payment, which is a very unstable kind of market.  If it becomes public that enough authors are unwilling to pay, our moral compass becomes activated and we start to question why we should pay (even a discounted price) when others are given a free ride.  The only other option is to keep waiver information secret and give authors the sense that everyone (or nearly everyone) is paying their fair share.

The second assumption — that the ability and willingness of future authors to pay publication charges remains constant — is based on the notion that future authors will be demographically similar now that PLoS ONE has been given an laudable impact factor. Adding proportionately more non-paying and under-paying authors puts PLoS ONE in a similar situation as raising the acceptance bar would — the cost of running their operation will go up.  This means either raising the price to those still willing to pay or reducing surplus payments to support PLoS flagship journals.  Either solution is gloomy.  The darkest outcome is a total market failure, where price sensitivity starts driving authors out of the payment group and into the free group, creating spiraling costs for the remaining and dwindling few.

In sum, PLoS ONE‘s first and astounding 4.35 impact factor should neither be reason to celebrate nor ridicule, for it puts the publisher in a position that may require them to start treating PLoS ONE like other conventional journals.

This may be time to consider launching PLoS TWO.

Enhanced by Zemanta
 

Jun 9, 2010

Impact Factors — A Self-fulfilling Prophecy?

Posted by Philip Davis under Authority, Metrics and Analytics, Research | Tags: , , , |
[8] Comments 
An apple and an orange.

Image via Wikipedia

Impact factors are both loved and despised.

Some argue they reflect little more than the entrenched hegemony of the established science journals conveying their prestige on the select few articles allowed to pass through their hallowed editorial gates. Others argue that articles are cited on their scientific merit alone and that citations have nothing to do with the source in which they are published.

The problem with understanding how much prestige a journal conveys on its articles is that journals are in the business of publishing unique articles.  Comparing the citation performance of articles published in one source against the next always comes down to comparing apples and oranges.

One way of getting around the apples to oranges problem is by seeking out cases of unethical multiple publication — identical (or nearly identical) articles published in multiple sources.  Another way is to track the performance of articles meant to be published in several sources.

In an article published this month in the Journal of Clinical Epidemiology, Thomas Perneger, a Swiss epidemiologist, tackles the question of impact factors and citations by analyzing the performance of medical consensus statements — public reports published simultaneously in multiple journals. Perneger focuses on just four statements published 33 times:

  • QUOROM (QUality Of Reporting Of Meta-analyses) – published 3 times
  • CONSORT (CONsolidated Standards Of Reporting Trials) revised – 8 times
  • STARD (STAndards of Reporting of Diagnostic accuracy) – 14 times
  • STROBE (STrenghening of the Reporting of OBservational studies in Epidemiology) – 8 times

By comparing the citation performance of these statements compared to the journal’s impact factor, Perneger hoped to investigate the degree to which a journal’s prestige is conveyed on its articles.  By using exact copies of articles, he was able to control for article characteristics (e.g. topic, authors, length) as well as variation in article quality, the bugaboo of most impact factor studies.

Not only were citation performances of the consensus statements highly correlated with their journal’s impact factor, Perneger reports, their relationships were linear — each logarithm unit increase in impact factor predicted one logarithm unit increase in citations, meaning that if one journal’s impact factor was twice that of another, you could predict a doubling of article citations. Perneger discusses the meaning of these results:

. . . the analysis presented here suggests that the association between journal impact factors and future citations is in part self-fulfilling prophecy, as publication in a high-impact journal will push upward the citations of any given article, independent of its value. This creates an additional but perverse incentive to pursue publication in high-impact journals.

Perneger’s study, while admittedly small in size, adequately addresses article to article differences. His set-up is simple and elegant; his is results are straightforward and intuitive.

Case closed?

Not so fast.

Perneger demonstrates a correlation between article citations and the prestige (as measured by impact factor) of the journal. If prestige were driving an author’s decision to cite one version of a consensus statement over another, the author would need to know that each version of the article existed in the first place, and then make a conscious decision to cite based on the prestige of the source. For instance, an author required to cite the STARD statement would need to know that it was published in 14 different sources, have an intuitive sense of the pecking order of the journals and select accordingly.

Does this sound reasonable?

The main weakness of this paper is the inability to separate the citing process from the information discovery process.  Highly cited journals tend to have higher circulation and broader readership in the scientific community.  If a consensus statement was published in a multidisciplinary medical journal like JAMA, The Lancet, or BMJ, it’s likely to get much more readership than if it were published in a smaller, specialized journal. Likewise, someone who does a general web search for the statement is likely to land on the larger and more prestigious of the journals.

In other words, it is just as plausible that Perneger’s results are explained by enhanced visibility than by prestige alone.

Until we get into the minds of the authors while they are preparing their manuscripts, we may never really know why one article gets cited over another . . . although it’s still fun trying.

—

Perneger, Thomas V. 2010. Citation analysis of identical consensus statements revealed journal-related bias. Journal of Clinical Epidemiology 63: 660-664, DOI: 10.1016/j.jclinepi.2009.09.012

Reblog this post [with Zemanta]
 

Jun 7, 2010

Forgetting the F-bomb: How Cheap Memory Makes Normal Conference Chat Costly

Posted by Philip Davis under Controversial Topics, Sociology, Technology | Tags: , , , |
[7] Comments 
The f-bomb via. Legaljuice

The f-bomb via. Legaljuice

At the closing SSP conference plenary, Food Fight in the Scholarly Kitchen, I dropped the “f-bomb.”  I call it the f-bomb because FTC rules prohibit me from using the real expletive, but the readers will know what I mean, and listeners in that session will remember what I said.

In the context of the question (about watchdog journalism) and my response (about taking risks as a blogger even though it might result in a frivolous lawsuit), the use of the “f-bomb” was, at the time, rather benign.  And yet, I have regrets.

My regrets are not founded in the memories of my fellow panelists or even in those of the audience participants.  Most of them will ultimately forget the remark, or more likely, not take the remark as being offensive.  In the context of the presentation, it was a rather normal event.

My regrets lay in the fact that the session was recorded for future posterity.  In a few weeks, the recording will show up on the SSP website.  It will be downloaded by robot software crawling the web, indexed and tagged with the participant names, and cached on multiple servers worldwide, thus ensuring that deletion of the original file does not hinder the preservation of the f-word remark.

What went on in San Francisco will not remain in San Francisco.

Thanks to the persistent memory of Google and other networked search and storage devices, I will carry that f-bomb into the future.  It will become part of my professional identity, and more importantly, a potential liability.  [In comparison, events that took place at the Fastiggi reception on Friday night remain in San Francisco, with the exception of a few carefully staged photographs].

Humans are wired to forget: the Web is wired to remember.  We have changed the default from forgetting to persistent memory.

This is the premise of the book, “Delete: The Virtue of Forgetting in the Digital Age” by Viktor Mayer-Schönberger (Princeton University Press, 2009).  Mayer-Schönberger argues that forgetting fulfills important societal functions.  As the details and consequences of our past mistakes fade and become obscured in the memories of those around us,  forgetting allows for individuals to reinvent themselves and move on.

But this is not an argument that we often hear.

We are told to save everything that is digital — all of our email, documents and memos (however mundane) — in the rare case that we would need to go back and retrieve some piece of old information.  Because storage is cheap, it costs us more in time and energy to purge old documents than it’s worth.  The default is to save everything.

Technology has made us all into compulsive hoarders.  We have become the hermit living in a one-bedroom apartment stuffed with every letter ever received and all issues of the New York Times piled up high into columns allowing only a thin canyon of passage to the kitchen sink and another to the bed.

And yet, there is a big difference.  When that pack rat moves out of the cramped and condemned apartment, relatives come in and clean house, transferring most items into the dumpster and saving a few mementos for posterity.  If the pack rat has no living relatives, the building superintendent is stuck with the chore of cleaning house.  Items will be relegated to the dustbin of history because physical space is not cheap like digital space.

The Internet does not clean house.  Even when websites close shop and lose their domain name, there is likely a digital copy sitting on the virtual shelves of Brewster Kahle’s Internet Archive.  Even when we want to purge our past, it may not be possible.

I like going to conferences like SSP precisely for the reason that the majority of the communication among peers takes place informally, in a non-seated position, without the harsh glare of a screen, disconnected from the Internet, and (after 5pm) with a drink in one’s hand.  These meetings enable us to say things that we dare not say in email or on a public blog.  If, by accident, the line of public decorum has been crossed we can be thankful that listeners understand the context (“He was drunk, and come to think of it, I was too!”) and that these events will gradually fade in the minds of colleagues.

“If we had to worry that any information about us would be remembered for longer than we live, would we still express our views on matters of trivial gossip, share personal experiences, make various political comments, or would we self-censor?  The chilling effect of perfect memory alters our behavior” (Mayer-Schönberger, p.5)

We like to think that conferences are doing non-participants a favor when they record and post their events.  We forget that by doing so, we may be harming the very reason why we spend enormous amounts of time and money traveling to conferences, which is, to allow for the kind of open and earnest dialogue that is becoming more difficult to do online.

What went on in San Francisco should stay in San Francisco, the occasional f-bomb included.

Reblog this post [with Zemanta]
 

Next Page »