Dan Is So Right About Experts
When it comes to relying on experts, Dan is absolutely right. Experts knowlege makes them uniquely qualified to predict the future and therefore we should heed their advice. Why just take a look at this long term study examining the proficiency of experts:
Tetlock got a statistical handle on his task by putting most of the forecasting questions into a “three possible futures” form. The respondents were asked to rate the probability of three alternative outcomes: the persistence of the status quo, more of something (political freedom, economic growth), or less of something (repression, recession). And he measured his experts on two dimensions: how good they were at guessing probabilities (did all the things they said had an x per cent chance of happening happen x per cent of the time?), and how accurate they were at predicting specific outcomes. The results were unimpressive. On the first scale, the experts performed worse than they would have if they had simply assigned an equal probability to all three outcomes—if they had given each possible future a thirty-three-per-cent chance of occurring. Human beings who spend their lives studying the state of the world, in other words, are poorer forecasters than dart-throwing monkeys, who would have distributed their picks evenly over the three choices.
Tetlock also found that specialists are not significantly more reliable than non-specialists in guessing what is going to happen in the region they study. Knowing a little might make someone a more reliable forecaster, but Tetlock found that knowing a lot can actually make a person less reliable. “We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” he reports. “In this age of academic hyperspecialization, there is no reason for supposing that contributors to top journals—distinguished political scientists, area study specialists, economists, and so on—are any better than journalists or attentive readers of the New York Times in ‘reading’ emerging situations.” And the more famous the forecaster the more overblown the forecasts. “Experts in demand,” Tetlock says, “were more overconfident than their colleagues who eked out existences far from the limelight.”
By the way, these experts are the same people that are constantly used by politicians to justify increasing state power. A tactic many of you support when you vote for politicians that favor expanding state government on the advisement of experts.
March 27th, 2009 at 12:04 pm
So I’m probably better at forecasting than these so-called experts? sweet.
March 27th, 2009 at 3:07 pm
Now Tetlock is an expert on the fallibility of experts. Interesting line of work.
March 27th, 2009 at 3:51 pm
Thanks Mitch for pointing out the irony in using an expert to debunk experts.
March 27th, 2009 at 4:05 pm
WHAT ARE WE GOING TO DO? EXPERTS ARE IMMUNE TO DEBUNKERY.
March 27th, 2009 at 4:13 pm
Torches and pitchforks?
March 28th, 2009 at 11:03 am
“In one study, college counsellors were given information about a group of high-school students and asked to predict their freshman grades in college. The counsellors had access to test scores, grades, the results of personality and vocational tests, and personal statements from the students, whom they were also permitted to interview. Predictions that were produced by a formula using just test scores and grades were more accurate.”
Humans are biased and flawed, this just highlights the importance of empirical testing and rigorous mathematical modeling.
Its also worth noting that he divides people into two categories:
1: “A hedgehog is a person who sees international affairs to be ultimately determined by a single bottom-line force… Whatever it is, the big idea, and that idea alone, dictates the probable outcome of events.”
2: “Foxes, on the other hand, don’t see a single determining explanation in history.”
It turns out that this effect is driven by Hedgehog experts, who performed atrociously (fox experts performed fairly well).
This reminds me of an earlier conversation we had -
http://enableate.com/steve/2008/1056
As for the business about experts being used to increase state power - experts are NOT needed for that.
http://en.wikipedia.org/wiki/Reichstag_fire
At most, they are one of several tools that could potentially be employed.
One final point is that science (especially hard science) is highly self-correcting, making it (over the long run) much more resistant to hedgehog thinking. For example, whether or not atoms existed was once a controversial question, but it would be difficult today to find a serious physicist who doubts the existence of atoms. It’s extremely difficult for expert A to maintain that atoms do not exist once expert B has designed an paradigm that allows direct observation of them.