I learned about the Open Syllabus Project in The New York Times in a piece by the project’s founders, Joe Karaganis and David McClure. It’s fascinating. The authors set out to scrape college Web sites and have put together the metadata for over 1 million syllabuses. You can find the open service they have put online here. The project identifies all of the materials assigned to undergraduates and ranks them by frequency. It’s still in beta. More information is forthcoming, but like many projects of this kind, it is being put out into the broader community for comment and further exploration. I am eager to see where this will lead. A new dataset can lead to new insights, many of which were not anticipated by the people who initiated the project.
The authors make a point about altmetrics and how this dataset and analysis could lead to a better understanding of the impact of publications. The “standard” metric, Journal Impact Factor (JIF), is no stranger to readers of the Kitchen, of course, and its many limitations have been rehearsed endlessly (including the differences between STM and HSS fields and between journals and books). When a scholar cites another, that tells us something; but it also says something about a work when it becomes adopted in course after course around the country; and it says more and different things when it is used in classrooms in different courses and even different fields. Play with the service a bit and you will be intrigued. Daniel Bell’s The Cultural Contradictions of Capitalism appears on 69 syllabuses; Barbara Kingsolver’s The Poisonwood Bible appears on 265. There is some noise in the data: Kingsolver’s book has a second listing, which would add another 12 instances of classroom usage. That will get scrubbed in time. It’s probably fair to say that the data is highly suggestive but not definitive. Metrics mavens will want more.
For my part, I continue to be puzzled as to why JIF is such a source of controversy. Of course there are different metrics, and some of the new metrics are very good indeed at identifying some impacts. The question is what you are measuring for and to whom. The value of JIF lies primarily in the fact that it is an administrative shorthand used by promotion committees. It has value in itself, but its outsized value lies outside the journals themselves and their publishers (who of course are blamed for everything). There are many currencies in the world, but what do you do when you pull into a gas station and they only take American? Different metrics measure different things for different audiences. A new thing, a new metric, can have value without negating the things that came before it. I am reminded of TED founder Richard Saul Wurman’s remark that we live in “the age of also.” Thus we can have JIF and also a count of Twitter followers, Facebook “likes,” and a number derived from how often a text is used in a classroom. And I would add another metric, now maligned by one and all: How many copies did it sell?
I suspect, though, that the greatest value of The Open Syllabus Project is not in its assertion of a new metric, and certainly not because it is open, but because it gathers a great deal of information together in one place, where it can be analyzed. Book publishers who don’t have access to this data otherwise (the big college textbook publishers all have proprietary databases of course adoptions) will find some of the information to be eye-opening. For example, I would bet that Kingsolver’s publisher has no idea how many courses assign her books; and that same publisher probably knows the identity of only a handful of the instructors who require it. Extending this line of thinking, the publisher of a book that is similar to Kingsolver’s in some ways may want to find out who is using Kingsolver in the classroom. The syllabus database, in other words, can measure a form of impact, but it can also serve as a marketing database.
Other uses will be made of this data. Let’s unleash the data scientists, and while we are at it, let’s aim to enrich the data and to make it fully compliant with the requirements of text and data mining. Kudos to Karaganis and McClure for this. Now let’s see what happens next.