What Expert?!

Here's an excerpt from this article about experts:

The book is Expert Political Judgment: How Good Is It? How Can We Know? by Philip E. Tetlock, a professor at the University of California at Berkeley. It summarizes the results of a truly amazing research project: Over seven years Tetlock got a wide range of experts and nonexperts to answer carefully constructed questions about the likelihood of specific future events. Tetlock then cranked all those numbers through every kind of statistical thresher, flail, and grinder you can imagine, and the result was clear: Experts don't actually exist. Specifically, experts were no better than nonexperts at predicting the future. They weren't even as good as computer programs that merely extrapolate the past. The best experts could not explain more than 20% of the variability in outcomes, but crude algorithms could explain 25% to 30%, and sophisticated algorithms could explain 47%. Consider what this means. On all sorts of questions you care about--Where will the Dow be in two years? Will the federal deficit balloon as baby-boomers retire?--your judgment is as good as the experts'. Not almost as good. Every bit as good.

Which is not to say that experts are no different from you and me. They're very different. For example, they're much more confident in their predictions than nonexperts are, though they obviously have no reason to be. For example, the members of the American Political Science Association predicted in August 2000 that a Gore victory was a slam dunk.

Experts can also give far more reasons for their predictions than nonexperts can. Their vast erudition lets them explain at daunting length why something will or won't happen. Not that all those reasons make the forecasts one bit better.