Superforecasting by Philip E. Tetlock and Dan Gardner
Superforecasting by Philip E. Tetlock and Dan Gardner
cha
seac
opyof
Super
for
eca
sti
ng
a
toneoft
hes
eret
ail
ers
:
praise for
SUPERFORECASTING
“Superforecasting is a rare book that will make you smarter and wiser.
One of the giants of behavioral science reveals how to improve at
predicting the future.”
— A D A M G R A N T,
New York Times bestselling author of Give and Take
“Good judgment and good forecasting are rare, but they turn out
to be made of teachable skills. By forcing forecasters to compete,
Tetlock discovered what the skills are and how they work, and this
book teaches the ability to any interested reader.”
— S T E WA R T BR A ND ,
president, The Long Now Foundation
“There isn’t a social scientist in the world I admire more than Phil
Tetlock.”
— T IM H A R F O R D ,
author of The Undercover Economist
ISBN 978‑0‑8041‑3669‑3
eISBN 978‑0‑8041‑3670‑9
2 4 6 8 10 9 7 5 3 1
First Edition
An Optimistic Skeptic
W
e are all forecasters. When we think about changing jobs, get-
ting married, buying a home, making an investment, launch-
ing a product, or retiring, we decide based on how we expect the
future will unfold. These expectations are forecasts. Often we do our
own forecasting. But when big events happen—markets crash, wars
loom, leaders tremble—we turn to the experts, those in the know.
We look to people like Tom Friedman.
If you are a White House staffer, you might find him in the
Oval Office with the president of the United States, talking about
the Middle East. If you are a Fortune 500 CEO, you might spot him
in Davos, chatting in the lounge with hedge fund billionaires and
Saudi princes. And if you don’t frequent the White House or swanky
Swiss hotels, you can read his New York Times columns and bestsell-
ing books that tell you what’s happening now, why, and what will
come next.1 Millions do.
Like Tom Friedman, Bill Flack forecasts global events. But there
is a lot less demand for his insights.
For years, Bill worked for the US Department of Agriculture in
Arizona—“part pick-and-shovel work, part spreadsheet”—but now
he lives in Kearney, Nebraska. Bill is a native Cornhusker. He grew
up in Madison, Nebraska, a farm town where his parents owned and
published the Madison Star-Mail, a newspaper with lots of stories
about local sports and county fairs. He was a good student in high
school and he went on to get a bachelor of science degree from the
University of Nebraska. From there, he went to the University of
to the forecasters who help us make decisions that matter far more
than any baseball game, we’re content to be ignorant.3
In that light, relying on Bill Flack’s forecasts looks quite reason-
able. Indeed, relying on the forecasts of many readers of this book
may prove quite reasonable, for it turns out that forecasting is not a
“you have it or you don’t” talent. It is a skill that can be cultivated.
This book will show you how.
I also didn’t mind because the joke makes a valid point. Open
any newspaper, watch any TV news show, and you find experts who
forecast what’s coming. Some are cautious. More are bold and confi-
dent. A handful claim to be Olympian visionaries able to see decades
into the future. With few exceptions, they are not in front of the
cameras because they possess any proven skill at forecasting. Accu-
racy is seldom even mentioned. Old forecasts are like old news—soon
forgotten—and pundits are almost never asked to reconcile what
they said with what actually happened. The one undeniable talent
that talking heads have is their skill at telling a compelling story with
conviction, and that is enough. Many have become wealthy peddling
forecasting of untested value to corporate executives, government of-
ficials, and ordinary people who would never think of swallowing
medicine of unknown efficacy and safety but who routinely pay for
forecasts that are as dubious as elixirs sold from the back of a wagon.
These people—and their customers—deserve a nudge in the ribs. I
was happy to see my research used to give it to them.
But I realized that as word of my work spread, its apparent mean-
ing was mutating. What my research had shown was that the average
expert had done little better than guessing on many of the political
and economic questions I had posed. “Many” does not equal all. It
was easiest to beat chance on the shortest-range questions that only
required looking one year out, and accuracy fell off the further out ex-
perts tried to forecast—approaching the dart-throwing-chimpanzee
level three to five years out. That was an important finding. It tells
us something about the limits of expertise in a complex world—and
the limits on what it might be possible for even superforecasters to
achieve. But as in the children’s game of “telephone,” in which a
phrase is whispered to one child who passes it on to another, and
so on, and everyone is shocked at the end to discover how much it
has changed, the actual message was garbled in the constant retell-
ing and the subtleties were lost entirely. The message became “all
THE SKEPTIC
To understand the “skeptic” half of that label, consider a young Tu-
nisian man pushing a wooden handcart loaded with fruits and veg-
etables down a dusty road to a market in the Tunisian town of Sidi
Bouzid. When the man was three, his father died. He supports his
family by borrowing money to fill his cart, hoping to earn enough
selling the produce to pay off the debt and have a little left over. It’s
the same grind every day. But this morning, the police approach the
man and say they’re going to take his scales because he has violated
some regulation. He knows it’s a lie. They’re shaking him down. But
he has no money. A policewoman slaps him and insults his dead fa-
ther. They take his scales and his cart. The man goes to a town office
to complain. He is told the official is busy in a meeting. Humiliated,
furious, powerless, the man leaves.
He returns with fuel. Outside the town office he douses himself,
lights a match, and burns.
Only the conclusion of this story is unusual. There are countless
poor street vendors in Tunisia and across the Arab world. Police cor-
ruption is rife, and humiliations like those inflicted on this man are
a daily occurrence. They matter to no one aside from the police and
their victims.
But this particular humiliation, on December 17, 2010, caused
Mohamed Bouazizi, aged twenty-six, to set himself on fire, and
Bouazizi’s self-immolation sparked protests. The police responded
with typical brutality. The protests spread. Hoping to assuage the
public, the dictator of Tunisia, President Zine el-Abidine Ben Ali,
visited Bouazizi in the hospital.
Bouazizi died on January 4, 2011. The unrest grew. On January
14, Ben Ali fled to a cushy exile in Saudi Arabia, ending his twenty-
three-year kleptocracy.
The Arab world watched, stunned. Then protests erupted in
Egypt, Libya, Syria, Jordan, Kuwait, and Bahrain. After three de-
cades in power, the Egyptian dictator Hosni Mubarak was driven
from office. Elsewhere, protests swelled into rebellions, rebellions into
civil wars. This was the Arab Spring—and it started with one poor
man, no different from countless others, being harassed by police, as
so many have been, before and since, with no apparent ripple effects.
It is one thing to look backward and sketch a narrative arc, as I
did here, connecting Mohamed Bouazizi to all the events that flowed
out of his lonely protest. Tom Friedman, like many elite pundits, is
skilled at that sort of reconstruction, particularly in the Middle East,
which he knows so well, having made his name in journalism as a
Arab Spring might never have happened, at least not when and as it
did, if the police had just let Mohamed Bouazizi sell his fruits and
vegetables that morning in 2010.
Edward Lorenz shifted scientific opinion toward the view that
there are hard limits on predictability, a deeply philosophical ques-
tion.4 For centuries, scientists had supposed that growing knowl-
edge must lead to greater predictability because reality was like a
clock—an awesomely big and complicated clock but still a clock—
and the more scientists learned about its innards, how the gears
grind together, how the weights and springs function, the better they
could capture its operations with deterministic equations and predict
what it would do. In 1814 the French mathematician and astronomer
Pierre-Simon Laplace took this dream to its logical extreme:
THE OPTIMIST
But it is one thing to recognize the limits on predictability, and quite
another to dismiss all prediction as an exercise in futility.
why we know that one-and two-day forecasts are typically quite ac-
curate while eight-day forecasts are not. With these analyses, meteo-
rologists are able to sharpen their understanding of how weather works
and tweak their models. Then they try again. Forecast, measure, re-
vise. Repeat. It’s a never-ending process of incremental improvement
that explains why weather forecasts are good and slowly getting better.
There may be limits to such improvements, however, because weather
is the textbook illustration of nonlinearity. The further out the fore-
caster tries to look, the more opportunity there is for chaos to flap its
butterfly wings and blow away expectations. Big leaps in computing
power and continued refinement of forecasting models may nudge the
limits a little further into the future but those advances gradually get
harder and the payoffs shrink toward zero. How good can it get? No
one knows. But knowing the current limits is itself a success.
In so many other high-stakes endeavors, forecasters are groping
in the dark. They have no idea how good their forecasts are in the
short, medium, or long term—and no idea how good their forecasts
could become. At best, they have vague hunches. That’s because the
forecast-measure-revise procedure operates only within the rarefied
confines of high-tech forecasting, such as the work of macroecono-
mists at central banks or marketing and financial professionals in
big companies or opinion poll analysts like Nate Silver.7 More often
forecasts are made and then . . . nothing. Accuracy is seldom deter-
mined after the fact and is almost never done with sufficient regular-
ity and rigor that conclusions can be drawn. The reason? Mostly it’s a
demand-side problem: The consumers of forecasting—governments,
business, and the public—don’t demand evidence of accuracy. So
there is no measurement. Which means no revision. And without
revision, there can be no improvement. Imagine a world in which
people love to run, but they have no idea how fast the average person
runs, or how fast the best could run, because runners have never
agreed to basic ground rules—stay on the track, begin the race when
we beat the control group by 78%. GJP also beat its university-
affiliated competitors, including the University of Michigan and
MIT, by hefty margins, from 30% to 70%, and even outperformed
professional intelligence analysts with access to classified data. After
two years, GJP was doing so much better than its academic competi-
tors that IARPA dropped the other teams.10
I’ll delve into details later, but let’s note two key conclusions that
emerge from this research. One, foresight is real. Some people—
people like Bill Flack—have it in spades. They aren’t gurus or oracles
with the power to peer decades into the future, but they do have a
real, measurable skill at judging how high-stakes events are likely to
unfold three months, six months, a year, or a year and a half in ad-
vance. The other conclusion is what makes these superforecasters so
good. It’s not really who they are. It is what they do. Foresight isn’t a
mysterious gift bestowed at birth. It is the product of particular ways
of thinking, of gathering information, of updating beliefs. These
habits of thought can be learned and cultivated by any intelligent,
thoughtful, determined person. It may not even be all that hard to
get started. One result that particularly surprised me was the effect
of a tutorial covering some basic concepts that we’ll explore in this
book and are summarized in the Ten Commandments appendix. It
took only about sixty minutes to read and yet it improved accuracy
by roughly 10% through the entire tournament year. Yes, 10% may
sound modest, but it was achieved at so little cost. And never forget
that even modest improvements in foresight maintained over time
add up. I spoke about that with Aaron Brown, an author, a Wall
Street veteran, and the chief risk manager at AQR Capital Manage-
ment, a hedge fund with over $100 billion in assets. “It’s so hard to
see because it’s not dramatic,” he said, but if it is sustained “it’s the
difference between a consistent winner who’s making a living, or the
guy who’s going broke all the time.”11 A world-class poker player we
will meet soon could not agree more. The difference between heavy-
weights and amateurs, she said, is that the heavyweights know the
difference between a 60 ⁄40 bet and a 40 ⁄ 60 bet.
And yet, if it’s possible to improve foresight simply by measur-
ing, and if the rewards of improved foresight are substantial, why
isn’t measuring standard practice? A big part of the answer to that
question lies in the psychology that convinces us we know things
we really don’t—things like whether Tom Friedman is an accurate
forecaster or not. I’ll explore this psychology in chapter 2. For cen-
turies, it hobbled progress in medicine. When physicians finally ac-
cepted that their experience and perceptions were not reliable means
of determining whether a treatment works, they turned to scientific
testing—and medicine finally started to make rapid advances. The
same revolution needs to happen in forecasting.
It won’t be easy. Chapter 3 examines what it takes to test fore-
casting as rigorously as modern medicine tests treatments. It’s a big-
ger challenge than it may appear. In the late 1980s I worked out a
methodology and conducted what was, at the time, the biggest test of
expert political forecasting accuracy ever. One result, delivered many
years later, was the punch line that now makes me squirm. But an-
other discovery of that research didn’t receive nearly as much atten-
tion even though it was far more important: one group of experts
had modest but real foresight. What made the difference between the
experts with foresight and those who were so hopeless they dragged
the average down to the level of a dart-throwing chimp? It wasn’t
some mystical gift or access to information others didn’t have. Nor
was it any particular set of beliefs. Indeed, within a quite wide range
of views, what they thought didn’t matter. It was how they thought.
Inspired in part by that insight, IARPA created its unprece-
dented forecasting tournament. Chapter 4 is the story of how that
happened—and the discovery of superforecasters. Why are they so
good? That question runs through chapters 5 through 9. When you
meet them it’s hard not to be struck by how smart they are, so you
might suspect it’s intelligence that makes all the difference. It’s not.
They’re also remarkably numerate. Like Bill Flack, many have ad-
vanced degrees in mathematics and science. So is the secret arcane
math? No. Even superforecasters who are card-carrying mathemati-
cians rarely use much math. They also tend to be newsjunkies who
stay on top of the latest developments and regularly update their fore-
casts, so you might be tempted to attribute their success to spending
endless hours on the job. Yet that too would be a mistake.
Superforecasting does require minimum levels of intelligence,
numeracy, and knowledge of the world, but anyone who reads seri-
ous books about psychological research probably has those prereq-
uisites. So what is it that elevates forecasting to superforecasting? As
with the experts who had real foresight in my earlier research, what
matters most is how the forecaster thinks. I’ll describe this in detail,
but broadly speaking, superforecasting demands thinking that is
open-minded, careful, curious, and—above all—self-critical. It also
demands focus. The kind of thinking that produces superior judg-
ment does not come effortlessly. Only the determined can deliver it
reasonably consistently, which is why our analyses have consistently
found commitment to self-improvement to be the strongest predictor
of performance.
In the final chapters, I’ll resolve an apparent contradiction be-
tween the demands of good judgment and effective leadership,
respond to what I think are the two strongest challenges to my re-
search, and conclude—appropriately for a book about forecasting—
with a consideration of what comes next.
algorithms, and Big Data. At its core, the forecasting I study involves
subjective judgment: it is people thinking and deciding, nothing
more. Isn’t the time for such sloppy guesswork drawing to a close?
In 1954, a brilliant psychologist, Paul Meehl wrote a small book
that caused a big stir.12 It reviewed twenty studies showing that well-
informed experts predicting outcomes—whether a student would
succeed in college or a parolee would be sent back to prison—were
not as accurate as simple algorithms that added up objective indica-
tors like ability test scores and records of past conduct. Meehl’s claim
upset many experts, but subsequent research—now more than two
hundred studies—has shown that in most cases statistical algorithms
beat subjective judgment, and in the handful of studies where they
don’t, they usually tie. Given that algorithms are quick and cheap,
unlike subjective judgment, a tie supports using the algorithm. The
point is now indisputable: when you have a well-validated statistical
algorithm, use it.
This insight was never a threat to the reign of subjective judg-
ment because we so rarely have well-validated algorithms for the
problem at hand. It was just impractical for math to displace plain
old thinking—in 1954 and even today.
But spectacular advances in information technology suggest we
are approaching a historical discontinuity in humanity’s relation-
ship with machines. In 1997 IBM’s Deep Blue beat chess champion
Garry Kasparov. Now, commercially available chess programs can
beat any human. In 2011 IBM’s Watson beat Jeopardy! champions
Ken Jennings and Brad Rutter. That was a vastly tougher computing
challenge, but Watson’s engineers did it. Today, it’s no longer impos-
sible to imagine a forecasting competition in which a supercomputer
trounces superforecasters and superpundits alike. After that hap-
pens, there will still be human forecasters, but like human Jeopardy!
contestants, we will only watch them for entertainment.
So I spoke to Watson’s chief engineer, David Ferrucci. I was sure
that Watson could easily field a question about the present or past
like “Which two Russian leaders traded jobs in the last ten years?”
But I was curious about his views on how long it will take for Wat-
son or one of its digital descendants to field questions like “Will two
top Russian leaders trade jobs in the next ten years?”
In 1965 the polymath Herbert Simon thought we were only
twenty years away from a world in which machines could do “any
work a man can do,” which is the sort of naively optimistic thing
people said back then, and one reason why Ferrucci— who has
worked in artificial intelligence for thirty years—is more cautious
today.13 Computing is making enormous strides, Ferrucci noted. The
ability to spot patterns is growing spectacularly. And machine learn-
ing, in combination with burgeoning human-machine interactions
that feed the learning process, promises far more fundamental ad-
vances to come. “It’s going to be one of these exponential curves that
we’re kind of at the bottom of now,” Ferrucci said.
But there is a vast difference between “Which two Russian lead-
ers traded jobs?” and “Will two Russian leaders trade jobs again?”
The former is a historical fact. The computer can look it up. The
latter requires the computer to make an informed guess about the
intentions of Vladimir Putin, the character of Dmitri Medvedev,
and the causal dynamics of Russian politics, and then integrate that
information into a judgment call. People do that sort of thing all
the time, but that doesn’t make it easy. It means the human brain is
wondrous—because the task is staggeringly hard. Even with com-
puters making galloping advances, the sort of forecasting that su-
perforecasters do is a long way off. And Ferrucci isn’t sure we will
ever see a human under glass at the Smithsonian with a sign saying
“subjective judgment.”
Machines may get better at “mimicking human meaning,” and
thereby better at predicting human behavior, but “there’s a difference
between mimicking and reflecting meaning and originating mean-
ing,” Ferrucci said. That’s a space human judgment will always oc-
cupy.
In forecasting, as in other fields, we will continue to see human
judgment being displaced— to the consternation of white- collar
workers—but we will also see more and more syntheses, like “free-
style chess,” in which humans with computers compete as teams,
the human drawing on the computer’s indisputable strengths but
also occasionally overriding the computer. The result is a combina-
tion that can (sometimes) beat both humans and machines. To re-
frame the man-versus-machine dichotomy, combinations of Garry
Kasparov and Deep Blue may prove more robust than pure-human
or pure-machine approaches.
What Ferrucci does see becoming obsolete is the guru model
that makes so many policy debates so puerile: “I’ll counter your Paul
Krugman polemic with my Niall Ferguson counterpolemic, and
rebut your Tom Friedman op-ed with my Bret Stephens blog.” Fer-
rucci sees light at the end of this long dark tunnel: “I think it’s going
to get stranger and stranger” for people to listen to the advice of
experts whose views are informed only by their subjective judgment.
Human thought is beset by psychological pitfalls, a fact that has only
become widely recognized in the last decade or two. “So what I want
is that human expert paired with a computer to overcome the human
cognitive limitations and biases.”14
If Ferrucci is right—I suspect he is—we will need to blend
computer-based forecasting and subjective judgment in the future.
So it’s time we got serious about both.
Super
for
eca
sti
ng
a
toneoft
hes
eret
ail
ers
: