A recent paper in Communications of the ACM entitled, “Conference Paper Selectivity and Impact,” deals with a number of interesting and relevant concepts for publishers of scholarly materials. It’s also a very robust study, comparing conferences and journals between 1970 and 2005, for a total of 14,017 conference papers and 10,277 journal articles.
The authors, Jilin Chen and Joseph A. Konstan, addressed three questions:
- How does a conference’s acceptance rate correlate with the impact of its papers?
- How much impact do conference papers have compared to journal papers?
- To what extent does the impact of a highly selective conference derive from filtering (selectivity) vs. signaling (holding a posture of selectivity)?
Of course, the immediate concern is about generalizability. These are computing papers and journals, and the culture of computing is its own. Papers presented at conferences in medicine are usually embargoed and not cited until published in a journal, but computing conference papers are commonly cited, and the culture supports this. So, I think the second question above isn’t worth pushing too hard in a general way.
However, the other two questions remain of interest.
What the authors find is that a more selective conference attracts papers that garner more citations. That’s not too surprising. But the authors go further, and seek to find whether a conference’s reputation generates some of this advantage, either by dissuading marginal authors in the first place or by serving as an attractor for citations.
To get to this question, they normalized all the conferences to the top 10% of their papers and analyzed the cumulative citations. So, the most restrictive conferences, which started with acceptance rates of 10-15%, were pretty much left as-is, while only the top 10% of papers were included from a conference that initially accepted 60% of submissions. If there were no effect of reputation, then the top 10% of papers would be equivalent.
Lo and behold, the citation line looks almost identical to that for all papers — a steady downstairs set of bar graphs depending on the conference’s stringency — indicating that the reputation of the conference somehow drives citations.
What does this mean? Simply, it means that reputation matters — better submissions and more citations accrue to conferences and journals that people think are tougher to get into.