Malcolm Gladwell emerges infrequently with analyses that often change perceptions on issues we think are innocuous but turn out to be subtly powerful.

In the most recent issue of the New Yorker, Gladwell takes on the U.S. News & World Report’s college rankings, an ordered listing of universities that has evolved from a utilitarian service from a third-rate news weekly to become the cornerstone of a rankings business that has outlasted its initial host.

The U.S. News rankings use seven weighted variables to derive one set number for each academic institution:

1. Undergraduate academic reputation — 22.5%
2. Graduation and freshman retention rates — 20%
3. Faculty resources — 20%
4. Student selectivity — 15%
5. Financial resources — 10%
6. Graduation rate performance — 7.5%
7. Alumni giving — 5%

The problem Gladwell describes with the U.S. News rankings (and with other rankings, like Car and Driver auto rankings) is that they try to make the incomparable comparable — they’re comprehensive and attempt to apply the same measurements to qualitatively different entities. In short, they’re heterogeneous.

U.S. News is pretty cagey in deploying its rankings, cautioning parents and incoming freshmen to look at all the available data and not to rely on the rankings, while weaving in text like “These results also serve as a validation of the U.S. News Best Colleges rankings methodology that weights undergraduate academic reputation at 22.5 percent.”

Other problems exist even deeper in the U.S. News approach. For example, 1/5 of the score is based on the category “faculty resources.” What is this, exactly? The goal is to measure “engagement,” but the rankings have to rely on proxies to estimate engagement. According to U.S. News, there are six factors they look at to determine engagement — two class-size components (big classes vs. small classes), faculty salary, faculty degree attainment, student:faculty ratio, and the proportion of the faculty who are full-time.

But why do some of these matter? Does it matter to quality whether my philosophy professor has an ABD or a PhD or an MA? And does a more highly paid professor equate to a more engaged professor? Unless things have really changed since I went to college, there’s almost an inverse relationship there.

In fact, wealth seems to be driving the U.S. News rankings, and in a particular passage, Gladwell loses his cool:

Rankings are not benign. They enshrine very particular ideologies, and, at a time when American higher education is facing a crisis of accessibility and affordability, we have adopted a de-facto standard of college quality that is uninterested in both of those factors.

Gladwell’s article is well worth reading (Penn State will suddenly appear to be a most admirable university). It touches on ranking of all types — hospitals, law schools, business schools, and so forth. U.S. News has turned this into a real industry, but a flimsy one.

Ultimately, this is all another reminder of how odd it is that highly educated and educationally ambitious people seem to seek clarity through numbers that, when you pull back the veil, are very poor proxies of quality, predictors of value, or estimates of differentiation.

Yes, I’m looking at you, impact factor.