Important reminder: forecasting more than 10 years out (and often more than 5 years out) is nearly impossible. This book won’t teach you to become an oracle, but it can teach you to get much better at answering questions like “Will the Chinese Yuan be worth less in 3 months than it is today?” The people who are stellar at answering these kinds of questions are what Tetlock calls “superforecasters”.

Fun fact: If you’ve ever heard someone say “Research shows that most forecasters are less accurate than dart-throwing chimpanzees,” they’re referencing Tetlock’s work. His original conclusion was “Many forecasters we studied do no better than random chance,” but the chimpanzee thing stuck because it’s more interesting.

If you want to be right, try to prove yourself wrong

People open themselves up to blowups when they don’t consider viewpoints that contradict their viewpoint.

If you’re so confident that you’re right, why not consider other viewpoints? It shouldn’t matter if you are indeed correct. In fact, those other viewpoints (that are clearly wrong) should help illustrate why your viewpoint is correct.

…If those thoughts make you uncomfortable, perhaps you shouldn’t be so confident.
People who try to prove themselves wrong are wrong more often when it doesn’t matter, and right more often when it does.

Tetlock says their are two types of forecasters: hedgehogs (who know a lot about one thing) and foxes (who know a little about a lot of things). Foxes were unequivocally better forecasters, because they considered many perspectives and tried to prove themselves wrong. (The fox/hedgehog distinction comes from an essay by Isiah Berlin.)

The wrong side of maybe fallacy

People tend to think that if there’s a >50% chance something will happen (e.g. a 75% chance that Hillary will win the election) and it doesn’t happen, then the forecast was wrong. The only way to know if a forecast is wrong though is to turn back time and run many simulations – which is impossible. 

The best proxy for seeing if someone is a good forecaster is to look at many of their forecasts and see how often they’re correct on average. If you predict a 70% chance of rain 100 days in a row and it never rains, you technically could’ve been right every time, but more likely you’re a bad forecaster.

Use numbers, not phrases, to express levels of certainty

When making a forecast, don’t say “There’s a fair chance this will happen” or “This event is highly improbable”. These expressions can mean vastly different things to different people, and that can lead to disastrous consequences in high-stakes situations. Instead, use percentages to express your certainty. Here’s a table from Sherman Kent.

Certainty	The General Area of Possibility
100%	Certain
93% (give or take about 6%)	Almost certain
75% (give or take about 12%)	Probable
50% (give or take about 10%)	Chances about even
30% (give or take about 10%)	Probably not
7% (give or take about 5%)	Almost certainly not
0%	Impossible

People tend to balk at quantification like this. Many objections come from people trying to avoid responsibility. (“I said there was a fair chance – I’m not wrong!!”) One reasonable objection is that a percentage might imply certainty. It doesn’t and shouldn’t, so communicate this to the beneficiary of your analysis. Otherwise you might find yourself on the wrong side of maybe.

Inside view vs. outside view

Outside view: the general example of a situation
Inside view: the specific example of a situation

Eg. When trying to answer a question like “How likely is it that the Place family owns a dog?”, the best practice is to start with the outside view (in this case, “What percentage of Americans own a dog?”), then adjust your estimate up or down based on the inside view (the additional information you have above the most general case).

If we knew the Place family is wealthy, we might say “60% of wealthy families own dogs compared to 40% of all families, so 60% is our new baseline.” If we knew something qualitative like “Steve loves dogs”, we should absolutely adjust our guess upwards, but we’d have to be careful since that’s difficult to quantify.

Scope insensitivity

People often respond the same way to a problem, regardless of degree, if the various degrees evoke the same image in their mind.

Example: One group of people in Ontario was asked how much they would pay to help clean up a nearby lake. Another group of people, also from Ontario, was asked how much they would pay to help clean up all 26,000 lakes in the region. Both groups said, on average, $10.

Example 2: Three groups of people were asked how much they would pay to help clean up oil spills. The experimental condition was the number of birds they were told were affected by the oil spill. The first group 2,000, the second 20,000, and the third 200,000. All groups said, on average, $80.

The point is – if you want people to react more strongly, make the image in their head more vivid. Hearing “[Insert large number] ducks were affected by a large oil spill” doesn’t motivate people. My proposed alternatives: show pictures, tell a story, talk about second-order effects (that are also lurid), and of course – figure out what people want (i.e. what motivates that specific person).

Notably, superforecasters tend to be sensitive to scope. Asking two groups of superforecasters similar questions like “What is the probability that the Assad regime will fall in the next 3 months?” vs “What is the probability the Assad regime will fall in the next 6 months?” produced different answers (15% and 24%, when Tetlock and Kahneman asked these specific questions). Regular forecasters did not produce significantly different answers.

Fermi-izing

To answer a tough question, like “How many piano tuners are there in Chicago?”, break it down into the components you would need to know to answer the question. This method was created by Enrico Fermi, of the Fermi Paradox.

With the piano tuner question, you would need to know:

1. How many pianos are in Chicago?
2. How often do they need to get tuned?
3. Given (2), how many piano tuners does the market support?

Then you can break this down further and start guessing. Somehow, the errors in the guesses (unless they’re very large) seem to cancel out and get you a pretty close estimate.

Let’s try it.

1. How many pianos are in Chicago?
   1. Let’s start by guessing how many people are in Chicago. 2.5 million seems to be a good guess, if there are about 2 million people in my home city of Houston (and Houston is the 4th largest U.S. city to Chicago’s 3rd).
   2. Let’s say about 1/100 residences own a piano. We’ll bump it up to 2/100 to account for music halls, universities and schools, etc.
   3. Multiply to get 50,000 pianos.
2. How often do they need to get tuned?
   1. Blind guess. Twice a year?
   2. So that’s 100,000 piano tunings a year.
3. How many piano tuners does the market support?
   1. Let’s think about how many pianos a tuner could tune in a day. I’ll guess 3, assuming a roughly 2-hour tuning process, time to travel between clients, and an 8-hour work day.
   2. A piano tuner probably works, like most people, 50 weeks a year. 3 a day times 5 days a week is 15. 15 pianos a week times 50 weeks is 750 pianos a year.
   3. Dividing the number of piano tunings needed a year by the number of pianos tuned by each tuner gives 100,000 / 750 = 133.

The actual number (according to the book) was about 83, but Tetlock came up with 63 in his Fermi-izing. I’m farther off than him, but my guess is still decent. (I’ll admit I borrowed a lot of his analysis, but I still tried to do this process on my own, hence the different answer.)

Fermi used this process to hypothesize that there should be life besides us in the universe (based on what we know about the number of galaxies and how many of them are likely to have planets in the habitable zone). And yet there’s not, at least that we’ve observed. Thus, Fermi’s Paradox.

You can think of Fermi-izing as creating your own outside view when one isn’t available to you.

Final thoughts

There’s a ton of great stuff that I haven’t covered here to keep this from getting too long. Check out the book! Even if you have no interest in forecasting, the improvement in your ability to think critically is invaluable.