Volume Is Not Usability

Web metrics are often relied upon simultaneously for two things — commercialization of Web content (selling the page views and ad impressions, or selling site licenses) and analysis of usability issues or user preferences. Using traditional Web metrics for commercialization currently works pretty well (but this is an evolving area). Web 2.0 and new technologies (AJAX and Flash) are making the traditional measures less reliable. And the problem runs deeper, even on the commercialization front. One group has estimated that 75% of the data Web marketers collect are either inaccurate or misleading.

While volume-based metrics can help a publisher judge uptake of new features and interest in various content options, they don’t work well when publishers are seeking usability insights. Clickstreams may not be informative because ultimately, what can you discern? As Gerry McGovern posits, “task completion” is the measure you should deploy if you’re trying to quantify the usability of your site:

Example: A customer clicks on page A, then leaves after 1 minute. What does that mean? If they stayed for 3 minutes would that have been better? Why? Supposing the person who spends 3 minutes on the page finds it cluttered and full of verbiage?

Example: A customer clicks on page C, then clicks on page M, then goes back to page C, then leaves. What does that mean? Did they think they were going to get something on page M that they didn’t get? Or did they get what they needed on page M, and were simply using the Back button to navigate out of the site?

We need to radically simplify how we measure the success of our websites. Here’s how:

• Identify the top three tasks of your website
• Give these tasks to your customers and measure whether or not they are able to complete them.

If you’re a university, a top task should be to find a course. Observe potential students as they try to find a course on your website. If you’re a health website, finding out the symptoms for a particular disease is probably a top task.

Jakob Nielsen has an excellent entry on this, as well, using the term “success rate” instead of task completion.

Identifying these tasks is easier said than done. Getting everyone to agree upon a Web site’s top three tasks requires some hard thinking and good discussions. Users may disappoint us with the pedestrian nature of their needs. Is it finding and retrieving a PDF? Is it recovering a faintly remembered article from 5 years ago? Is it perusing the current issue’s contents? Or is it something else?

The biggest challenge in Web analytics continues to be asking the right questions. Only then do you know what data you need. Without the right questions, you can’t distinguish signal from noise.