Authority, Commerce, Experimentation, Metrics and Analytics, Social Media, Technology, Tools, World of Tomorrow

How Do You Solve a Problem Like Clay Shirky? Or, Silicon Valley Discovers Impact Factor

Image representing PeerIndex as depicted in Cr...

Image via CrunchBase

“If you are not paying for it, you’re not the customer; you’re the product being sold.”

So said Andrew Lewis aka blue beetle on a metafilter post about user driven discontent. I was thinking about this as I perused the latest Social Media (book marketing) YouTube video. And then, my Twitter feed coughed up an article from the New York Times entitled, “Got Twitter? You’ve been scored

Here’s the opening paragraph:

IMAGINE a world in which we are assigned a number that indicates how influential we are. This number would help determine whether you receive a job, a hotel-room upgrade or free samples at the supermarket. If your influence score is low, you don’t get the promotion, the suite or the complimentary cookies. This is not science fiction. It’s happening to millions of social network users.

It sounds awfully familiar, doesn’t it? A single number for  “influence?”

It’s the impact factor, of course. As we all know, these numbers really matter — to the highly cited goes the grant funding, the tenure, and the fortune and glory (presumably).  They matter to publishers as well, of course. Here in the UK, these numbers have been used by universities as they chase money from the four higher education national funding councils — research from full-time members of staff has been filtered to select the “best” outputs (for which impact factor has been a well-used proxy). These are long-term decisions. The 2008 Research Assessment Exercise (RAE) results determined funding to universities through 2013. Interestingly (especially in the light of the Times article), the successor to the RAE, the catchily named Research Excellence Framework, will be using some nebulous concept of “impact” in order to dole out the cash.

Back to Twitter, Facebook, and the rest. As we move into the “Me Centered Web,” certain parties are very interested in working out how to make money off the free tools that have been provided for us to connect and share with. We are the product being sold. Companies like Klout and PeerIndex are attempting to distill the complex swirling interactions of the global social network into some simple numbers assigned to each user, whether we like it or not. I’m ranked based on my Twitter activity — I’ve not signed up to any of these companies, though note this: PeerIndex told me which of my followers had signed up.

As the Times article states, typically the 200 million plus users of Twitter, or the 750 million users of Facebook, are sorted into a numbering system that ranks them between 1 and 100. It seems a little crude even if you add in a touch of semantic analysis to the texts/links of the tweets, or the users’ biographies. These metrics also seem ripe to be gamed, especially when you take into account the (virtual) rewards based incentives the new companies are using to entice users back in order to improve the money-spinning database of influential profiles.

There are other issues as well. If you are not particularly active online (enter Clay Shirky or perhaps Malcolm Gladwell), you don’t rank for influence. Now, whatever one may think of Clay Shirky and his writing, the facts are that he’s very smart, very thoughtful, and what he writes is usually very important reading, especially in the area of social interactions and information creation in the digital age. A metric that can’t cope with the creator of two of the key works on the impact of the Internet on human information exchange might be deemed to be in need of improvement.

To be fair, the CEO of PeerIndex acknowledges the point. These influence metrics seem to have little room for acts of significant content creation. This is not a knee jerk “short form communications are rubbish” argument, but nevertheless measuring Facebook updates, link swaps, and likes or tweets does seem to largely exclude creators of longer form content. It’s tempting to point these new companies in the direction of the eigenfactor and to tell them to go take a look at the article level metrics of PLoS ONE or the work of project MESUR — it’s not often that scholarly publishing leads Silicon Valley.

But before we get too smug, the impact factor can also be gamed, and it has plenty of critics. There has been a steady stream of initiatives looking at different ways of measuring the influence of the researcher in their chosen field. Mendeley, altmetrics.org, academia.edu, PLoS ONE’s article metrics; all are tilting at the processes that keep our current quality metric in place. The Beyond Impact initiative is looking to try and synthesise a more complete view of what constitutes scholarly impact.

Buried in these initiatives is the idea that impact, influence, and quality will be determined by more complex signal processing than the impact factor provides for the content container we call the journal. Why? Because the things that need to be measured will extend out beyond the journal article. Don’t believe me? Go check out the Research Excellence Framework again, with it’s requirement to demonstrate the economic and social impacts of basic scholarly research.

Klout, PeerIndex, Facebook, Twitter, and the rest are all trying to solve the next problem in information filtration, and then the problem of how to monetise the solution.

Back in 1998, Sergey Brin and Larry Page effectively solved the filter failure issue of the early Web. PageRank was a signal:noise filter that transformed the cacophony of  the Web. They monetised by matching search terms to advertising, connecting users to companies touting their wares. But now, the next wave is upon us. You can’t argue with the numbers. Society, or at least a major subset of it, has embraced the next iteration of the Web — peer-to-peer information exchange at scale. And as this latest method of communication grows, we experience once again the familiar sensations resulting from being on the receiving end of too much information — sensory processing disorder on a global scale.

This problem will be solved. The company that does it will likely bring a transformation at least as dramatic as Google did.

Scholarly publishing should take no comfort from the fact that scholars don’t use Web 2.0 tools as part of their work. Some of them do. In fact, 13% of them do if you agree with the research. Some of those users like tilting at windmills. We shouldn’t dismiss Klout, PeerIndex, or whoever as just another bunch of VC funded upstarts. We should pay close attention. Social networks are not a fad. They are a fundamental component of the new information infrastructure. And we will all be analysed within them, one way or another. Publishers can have a meaningful role to play here. Just take a look at project cascade from the New York Times R&D department:

From long-form content to online reputation and influence. Now that’s how to solve a problem like Clay Shirky.

Enhanced by Zemanta

Discussion

8 thoughts on “How Do You Solve a Problem Like Clay Shirky? Or, Silicon Valley Discovers Impact Factor

  1. Hmmm, sounds to me like someone read Cory Doctorow’s “Down and Out In The Magic Kingdom” and is trying to turn his concept of “whuffie” into a real world phenomenon:

    https://secure.wikimedia.org/wikipedia/en/wiki/Whuffie

    Then again, as the Wikipedia article above shows, it’s not a new concept, and dates back to science fiction from over 50 years ago.

    Posted by David Crotty | Jun 28, 2011, 7:17 am
  2. This article evokes some scary Orwellian visions of the future, where the “death panels” envisioned by Sarah Palin will decide our fates based on our “impact factor.”

    Posted by Sandy Thatcher | Jun 28, 2011, 8:44 am
  3. Not sure I fully understand the point, but Clay Shirkey and Malcolm Gladwell are very “influential” in their niches, but are they outside them? Very many people have never heard of them and live quite happily without ever coming into contact with their writings or musings. Whereas someone like Barak Obama is “influential” in any sphere.

    Posted by Maxine | Jun 28, 2011, 10:18 am
  4. Wonder whether these lovely graphics will move us from “all the news that’s fit to print” to “only the news likely to be tweeted.” We all want to provide our readers with important and relevant research but don’t we also still have the responsibility to record for the future meaningful (but not necessarily popular) additions to the body of knowledge.

    Posted by Judy Holoviak | Jun 30, 2011, 9:09 am
    • I actually got to see Cascade in action. The video doesn’t bring the analytical powers of the tool to life in the same way as actually watching a journalist using it as intended – a business intelligence tool to understand the propagation of their work through and beyond the NYTimes readership. Unlike, say, the AOL content farm franchises and their focus on clicks above all else, the Times was trying to understand the dynamics of the material they publish, in order to do it better in a digital world. As an example, Tech/Science news focus on a Thurs? Why? Is that an artefact of print scheduling?

      Your point about meaningful but not popular additions to the new, or indeed the scholarly corpus is absolutely spot on. I’m hoping that point was implicit in my argument – many discoveries are only deemed important and of high quality after they are rediscovered. Quality simply isn’t an intrinsic property. I’m sure we can all think of examples of information that we deemed important but which wasn’t picked up outside of niche outlets. But now we are heading into ‘filter failure’ and whether we are living in a information bubble territory…

      Posted by David Smith | Jun 30, 2011, 10:07 am
    • News business has always wanted to provide only that which is most popular for their chosen audience. It’s never been about “all the news”. Even if NYT decides to reduce the amount of news it spews out and focus only on the “best” however they define that, that will just leave more room for those serving the long tail.

      Posted by Matthew Thomas | Jul 7, 2011, 9:16 pm
  5. “If you are not particularly active online (enter Clay Shirky or perhaps Malcolm Gladwell), you don’t rank for influence.”

    Of course. This is a case where communication is taking a new route, or at least new branch. No system can hope to gather in all possible inputs. Some will be lost, even the valuable outliers. If ISI was around when the transition to journals began, critics of IF would have claimed that it didn’t satisfactorily recognize the differences in impact between books and articles and letters, etc.

    Posted by Matthew Thomas | Jul 7, 2011, 9:21 pm
  6. If fault is to be found with Shirky, as well as almost all other internet pundits on information overload, it is in their premises, not their conclusions. Almost all hold the implicit assumption that humans are sensitive to information as static facts. However, if informed by the most recent findings from affective neuroscience on human decision making, this position cannot be true.

    Specifically, Shirky (and nearly all of his peers) hold to positions that are not neurally realistic, and would have to abandon much of their opinions (and specifically the reality of information overload) if they were informed by the recent findings in affective neuroscience on how human minds actually process and choose information. Surprisingly, this argument can be made quite simply, and is made (link below) using an allegory of the Boston Red Sox pennant run over the years.

    http://mezmer.blogspot.com/2012/02/searching-for-red-stockings-myth-of.html

    (Alas, my argument at three pages is a bit long for a comments section, but perhaps not as a link.)

    A. J. Marr

    Posted by A Marr | Feb 25, 2012, 6:28 pm

Side Dishes by Stewart Wills

Find Posts by Category

Find Posts by Date

June 2011
S M T W T F S
« May   Jul »
 1234
567891011
12131415161718
19202122232425
2627282930  

The Scholarly Kitchen on Twitter

SSP_LOGO
The mission of the Society for Scholarly Publishing (SSP) is "[t]o advance scholarly publishing and communication, and the professional development of its members through education, collaboration, and networking." SSP established The Scholarly Kitchen blog in February 2008 to keep SSP members and interested parties aware of new developments in publishing.
......................................
The Scholarly Kitchen is a moderated and independent blog. Opinions on The Scholarly Kitchen are those of the authors. They are not necessarily those held by the Society for Scholarly Publishing nor by their respective employers.
Follow

Get every new post delivered to your Inbox.

Join 14,426 other followers

%d bloggers like this: