- Image via Wikipedia
A recent paper in Communications of the ACM entitled, “Conference Paper Selectivity and Impact,” deals with a number of interesting and relevant concepts for publishers of scholarly materials. It’s also a very robust study, comparing conferences and journals between 1970 and 2005, for a total of 14,017 conference papers and 10,277 journal articles.
The authors, Jilin Chen and Joseph A. Konstan, addressed three questions:
- How does a conference’s acceptance rate correlate with the impact of its papers?
- How much impact do conference papers have compared to journal papers?
- To what extent does the impact of a highly selective conference derive from filtering (selectivity) vs. signaling (holding a posture of selectivity)?
Of course, the immediate concern is about generalizability. These are computing papers and journals, and the culture of computing is its own. Papers presented at conferences in medicine are usually embargoed and not cited until published in a journal, but computing conference papers are commonly cited, and the culture supports this. So, I think the second question above isn’t worth pushing too hard in a general way.
However, the other two questions remain of interest.
What the authors find is that a more selective conference attracts papers that garner more citations. That’s not too surprising. But the authors go further, and seek to find whether a conference’s reputation generates some of this advantage, either by dissuading marginal authors in the first place or by serving as an attractor for citations.
To get to this question, they normalized all the conferences to the top 10% of their papers and analyzed the cumulative citations. So, the most restrictive conferences, which started with acceptance rates of 10-15%, were pretty much left as-is, while only the top 10% of papers were included from a conference that initially accepted 60% of submissions. If there were no effect of reputation, then the top 10% of papers would be equivalent.
Lo and behold, the citation line looks almost identical to that for all papers — a steady downstairs set of bar graphs depending on the conference’s stringency — indicating that the reputation of the conference somehow drives citations.
What does this mean? Simply, it means that reputation matters — better submissions and more citations accrue to conferences and journals that people think are tougher to get into.
Discussion
9 Thoughts on "Signals of Quality — Does Selectivity Attract Better Submissions and Better Authors?"
The crucial assumption that “If there were no effect of reputation, then the top 10% of papers would be equivalent” (in terms of citation) may not be true. For example, the best papers may only be submitted to the most restrictive conferences. In that case the restrictive conference papers get more citations because they are more important (what you call “better”, not because of the reputation of the conference.
PEER RANKING VS. PEER REVIEW
Chen & Konstan’s (C & K) paper is interesting, though (as noted) somewhat limited because it is based only on computer science and has fuller data on conference papers than on journal papers.
The finding is that papers from highly selective conferences are cited as much as (or even more than) papers from certain journals. (Journals of course also differ among themselves in quality and acceptance rates.)
Comparing the ceiling for citation counts for high- and low-selectivity conferences by analyzing only the equated top slice of the low-selectivity conferences, C & K found that that the high-selectivity conferences still did better, suggesting that the difference was not just selectivity but also “reputation.” (The effect broke down a bit in comparing the very highest and next-to-highest selectivity conferences, with the next-to-highest doing slightly better than the very highest; plenty of post hoc speculations were ready to account for this effect too: excess narrowness, distaste for competition, etc. at the very highest level, but not the next-highest…)
Some of this smacks of over-interpretation of sparse data, but I’d like to point out two rather more general considerations that seem to have been overlooked or under-rated:
(1) Conference selectivity is not the same as journal selectivity. A conference accepts the top fraction of its submissions (whatever it sets the cut-off point to be), with no rounds of revision, resubmission and re-refereeing (or at most one cursory final round, when the conference is hardly in the position to change most of its decisions, since the conference looms and the program cannot be made much more sparse than planned). Journals do not work this way; they have periodic issues, which they must fill, but they can have a longstanding backlog of papers undergoing revision that are not published until and unless they have succeeded in meeting the referee reports’ and editor’s requirements. The result is that the accepted journal papers have been systematically and dynamically improved through peer review (sometimes several rounds), whereas the conference papers have simply been statically ranked much as they were submitted. This is peer ranking, not peer review.
(2) “Reputation” really just means track record: How useful have papers in this venue been in the past? Reputation clearly depends on the reliability and validity of the selective process. But reliability and validity depend on more than the volume and cut-off point of raw submission rankings.
I normally only comment on open-access-related matters, so let me close by pointing out a connection:
There are three kinds of selectivity: journal selectivity, author selectivity and user selectivity. Journals (and conferences) select which papers to accept for publication; authors select which papers to submit, and which publication venue to submit them to; and users select which papers to read, use and cite. Hence citations are an outcome of a complex interaction of all three factors. The relevant entity for the user, however, is the paper, not the venue. Yes, the journal’s reputation will play a role in the user’s decision about whether to read a paper, just as the author’s reputation will; and of course so will the title and topic. But the main determinant is the paper itself. And in order to read, use and cite a paper, you have to be able to access it. Accessibility trumps all the other factors: it is not a sufficient condition for citation, but it is certainly a necessary one.
Existence is a pre-requisite for access, so you should be pro-wordprocessing because that trumps all other factors?
Kent,
Just read the paper and spoke to a computer scientist for confirmation.
While this is technically an interesting paper, its main contribution — that signaling contributes to citation performance — cannot be derived from their analysis.
It makes no sense to think that the top 10% of papers from each conference are comparable. The best of the worst simply cannot be compared to the best of the best. I don’t need to reiterate David and Stevan’s comments above.
The citation differences between conference papers and journal articles may be partly explained by barriers to entry. If your conference paper is accepted, you are required to travel to the conference (often held in an exotic location) and present it in person. This limits submission to computer scientists with access to funds for travel and lodging. In comparison, the barriers to entry for publishing a journal article are much, much lower.
Moreover, many computer science journal articles are merely expanded versions of conference proceeding papers published earlier.
In sum, this paper confirms two widely held beliefs: 1) that acceptance rates are inversely related to citation performance; and 2) that conference papers outperform journal articles. Their most interesting claim — that signaling is partially responsible for the citation advantage to prestigious conferences — cannot be demonstrated in their analysis.
Well, I disagree. Their supposition is that by moving the bar to an equivalent level of selectivity post-hoc, so that every conference becomes a 10% acceptance conference, you see the effect of other factors, one of which could be reputation. If filtering were the only factor, then citation counts to the top 10% of papers would be more similar. Citations per increased but the fit was significant. If not reputation, what else explains the fit?
The simple explanation is that better articles get submitted to more prestigious conferences.
This is self-selection and is the same process that operates in the journal publishing world.
Comparing the top 10% of each conference venue assumes that conference submission is a random or arbitrary process and that each conference has a similar and comparable sample of submissions — the only difference between these conferences being the stringency of the filtering process.
Its an assumption that is far from reality.
A post-hoc analysis based on citation performance can only tell you that higher cited papers appear in more prestigious outlets. It can’t allow you distinguish the effect of paper quality from the effect of journal prestige.
Just a quick note (and appreciation that our paper is generating such discussion).
When we wrote about the effects of signaling, we were including signaling at both ends (the attraction of authors as well as of writes). We completely agree that some of this effect is self-selection; some of it may also be post-hoc selection among readers/citers. We can’t separate the two.
The point about barriers to entry is an interesting one (it doesn’t much distinguished different CS conferences, but clearly distinguishes journals). A larger factor, though, is probably the external systems that place value on particular types of publishing. In most US Computer Science departments, a faculty member would not get much credit towards tenure or promotion for a paper published at a conference with an acceptance rate above 25% or 30%. In some places in Europe, it is still the case that anything not published in a journal doesn’t count for much. These are dynamic, of course, but both add to the complexity of the issue and the difficulty of generalizing beyond Computer Science.
“If not reputation, what else explains the fit?”
Another possible explanation would be that the more selective conferences also cover broader topics so that the papers presented there attract readers from a wider range of sub-specialties in the field, which in turn would lead to greater citations. I don’t know the field well enough to know if this is the case.
On a related note Nature just published a News special on science metrics:
http://www.nature.com/news/specials/metrics/index.html
