45x17 - Prediction by the Numbers

[bunniefuu]

NARRATOR: The future
unfolds before our eyes

but is it always
beyond our grasp?

What was once the
province of the gods

has now come
more clearly into view

through mathematics and data

Out of some early
observations about gambling

arose tools that guide

our scientific
understanding of the world

and more

through the power of prediction

[car speeding, alarm
blaring, waves crashing]

From our decisions
about the weather

NEWSCASTER: The strongest
hurricane ever on record

NARRATOR: To finding
someone lost at sea

COAST GUARD CREWMAN:
Commencing search pattern

Keep a good look out!

NARRATOR: Every day
mathematics and data combine

to help us envision
what might be

It's the best crystal ball
that humankind can have

NARRATOR: Take a trip
on the wings of probability

into the future

MONA CHALABI:
We're thinking about luck

or misfortune,

but they just basically are
a question of math, right?

"Prediction by the Numbers"...

Right now, on "NOVA"

♪♪

NARRATOR: The Orange County
Fair, held in Southern California

In theory, these crowds
hold a predictive power

that can have
startling accuracy,

but it doesn't belong to
any individual, only the group

And even then, it
has to be viewed

through the lens of mathematics

The theory is known as
the "wisdom of crowds,"

a phenomenon first documented
about a hundred years ago

Statistician Talithia
Williams is here

to see if the theory checks
out, and to spend some time

with the fair's most
beloved animal,

Patches, a 14-year-old ox

[snorts]

TALITHIA WILLIAMS: It
was a fair kind of like this one

where, in 1906,

Sir Francis Galton
came across a contest

where you had to guess
the weight of an ox,

like Patches you
see here behind me

♪♪

NARRATOR: After the ox
weight-guessing contest was over,

Galton took all the entries home
and analyzed them statistically

To his surprise,

while none of the individual
guesses were correct,

the average of all the guesses

was off by less than one percent

That's the wisdom of crowds

But is it still true?

So, here's how I think
we can test that today

What if we ask a random
sample of people here at the fair

if they can guess how many
jelly beans they think are in the jar

And then, we take those
numbers and average them

and see if that's actually close

to the true number
of jelly beans

♪♪

Guess how many
jelly beans are in here

Come on, guys, everybody's
got to have their guess

I see your mind churning





Probably like 925?

I think a thousand

WILLIAMS: So just
write your number down

Uh huh, there you go

Can I have a jelly bean?

[laughter]

NARRATOR: The 135 guesses
gathered from the crowd vary wildly

WILLIAMS: The range
of our guesses was,

from the smallest was


So you can tell, folks
were really guessing

But when we take the average
of our guesses, we get 1,522

So the question is,

how close is our average to
the actual number of jelly beans?

Well, now's the moment of truth

♪♪

All right, so the real number
of jelly beans was 1,676

The average of our guesses
was off by less than ten percent

So there actually was
some wisdom in our crowd

NARRATOR: Though
off by about ten percent,

the average of the
crowd's estimates

was still more accurate

than the vast majority
of the individual guesses

Even so, the wisdom of
crowds does have limits

It can be easily undermined
by outside influences

and tends to work best on
questions with clear answers,

like a number

The steps Talithia took

reflect a process going
on all around us these days

in the work of statisticians

Thanks, everybody

So we collected this data,

right, we analyzed
it mathematically,

and we got an estimate
that was pretty close

to the actual true value

That's math and
statistics at work

♪♪

NARRATOR: We didn't always use
math and statistics to make predictions

The Romans studied the
flights and cries of birds

The Chinese cracked "oracle"
bones with a hot metal rod

and read the results


Russians used chickens

[chicken clucking]

Throughout history,
we've sought the future

in moles on people's faces,

clouds in the sky,

or a pearl cast into an iron pot

And that list of things used
for predicting goes on and on

But more recently... that is
the last couple hundred years...

To see into the future,
we've turned to science

and made some
remarkable predictions

from the existence of Neptune,

or radio waves,

or black holes,

to the future location of a
comet with such precision

we could land a
space probe on it

[probe beeping]

But if you pop the
hood of science,

inside you'll find a field
of applied mathematics

that's made many of
those predictions possible:

statistics

Statistics is kind of unique

It's not an empirical science
itself, but it's not pure math,

but it's not philosophy either

It's the framework,
the language,

the rules by which we do science

From that, we can
make decisions,

we can make conclusions,
we can make predictions

That's what that's what
statisticians try to do

Why I love statistics is that

it predicts the likelihood
of future occurrences,

which really means
it's the best crystal ball

that humankind can have

♪♪

NARRATOR: Ultimately, all
the predictive power of statistics

rests on a revolutionary insight
from about 500 years ago...

That chance itself can be tamed

through the
mathematics of probability

Viva Las Vegas!

Here's a city full of palaces

built on understanding
probability

and fueled by gambling,

which may seem a funny place

to find mathematician
Keith Devlin

But mathematics and gambling

have been tied
together for centuries

Today in a casino,
you'll find roulette,

slot machines,

blackjack

Playing craps is also
known as "rolling the bones,"

which is more accurate
than you might think

KEITH DEVLIN: Humans
have been gambling

since the beginnings
of modern civilization

The ancient Greeks,
the ancient Egyptians,

would use the ankle bones of
sheep as a form of early dice

NARRATOR: Surprisingly, while
the Greeks laid the foundation

for our mathematics,
they didn't spend any effort

trying to analyze
games of chance

DEVLIN: It seems to have
never occurred to them,

or indeed to anybody way
up until the 15th, 16th century,

that you could apply mathematics

to calculate the way these
games would come out

♪♪

NARRATOR: 16th-century Italian
mathematician Gerolamo Cardano

made a key early observation:

that the more times a
game of chance is played,

the better
mathematical probability

predicts the outcome,

later proven as the
law of large numbers

Examples of the law of large
numbers at work surround us

DEVLIN: When I flip this coin,

we have no way of knowing

whether it's going to
come up heads or tails

♪♪

That time it was heads

On the other hand, if I
were to toss a coin 100 times,

roughly 50% of the time
it would come up heads,

and 50% of the time
it would come up tails

We can't predict a single toss

We can predict the aggregate
behavior over a 100 tosses

That's the law of large numbers

NARRATOR: In fact,
casinos are a testament

to the iron hand of the
law of large numbers

The games are designed

to give the casinos a
slight edge over the gambler

Take American roulette:

on the wheel are the
numbers one through 36,

half red and half black

Betting a dollar on
one color or the other

seems like a 50-50 proposition

But the wheel also has
two green slots with zeros

If the ball lands in those,

the casino wins all the
bets on either red or black

And that's the kind of edge

that makes the casino
money over the long run

Customers are gambling

The casino is
absolutely not gambling

Because they may lose money,

they may lose a lot of
money to one or two players,

but if you have thousands
and thousands of players,

by the law of large numbers,

you are guaranteed to make money

♪♪

NARRATOR: The
law of large numbers

comes into play
outside of gambling too

In basketball, a field goal
or sh**ting percentage

is the number of baskets made

divided by the
number of shots taken

But early in the season,

when it's based on a
low number of attempts,

that percentage
can be misleading

JORDAN ELLENBERG: At the beginning
of the season, a less skilled player

might get off a few
lucky shots in a row

And at that point,

they'd have a super-high
sh**ting percentage

♪♪

NARRATOR: Meanwhile,
a very skilled player

might miss a few at the
beginning of the season

and have a low
sh**ting percentage

[buzzer]

But as the season goes on,

and the total number
of shots climbs,

their sh**ting
percentages will soon reflect

their true skill level

[buzzers and chimes going off]

That's the law of
large numbers at work

A small sample, like just a
few shots, can be deceptive,

while a large sample,
like a lot of shots,

gives you a better picture

♪♪

The gambling observations that
led to the law of large numbers

were a start, but what really
launched probability theory

and opened up a door
to a whole new way

of thinking about the future,

was a series of letters

exchanged between two
French mathematicians,

Blaise Pascal and Pierre
de Fermat in the 1650s,

about another gambling problem
that had been kicking around

for a few centuries

A simplified version of
the problem goes like this:

two players... let's call
them Blaise and Pierre...

Are flipping a coin

Blaise has chosen
heads, and Pierre tails

The game is the best of 5 flips,

and each has put
money into the pot

They flip the coin three times,
and Blaise is ahead two to one

But then the game is interrupted

What is the fair
way to split the pot?

The question is: how
do they divide up the pot

so that it's fair to what
might have happened

if they'd been able
to complete the game

NARRATOR: Fermat suggested
imagining the possible future outcomes

if the game had continued

There are just
two more coin flips,

creating four
possible combinations

Heads-heads, heads-tails,
tails-heads, and tails-tails

In the first three, Blaise
wins with enough heads

Pierre only wins
in the last case,

so Fermat suggested
that a three-to-one split

was the correct solution

The key breakthrough

was imagining the
future, mathematically,

something even
Pascal had trouble with

Because what Fermat did was
say, "Let's look into the future,

"look at possible futures,

"and we'll count the way
things could have happened

in different possible futures"

It was a simple
arithmetic issue,

but the idea of counting
things in the future

was just completely new,

and Pascal couldn't
wrap his mind around it

NARRATOR: Eventually Pascal
accepted Fermat's solution,

as did others,

and today, that exchange
of letters is regarded

as the birth of modern
probability theory

People realized the
future wasn't blank

You didn't know exactly
what was going to happen,

but you could calculate
with great precision

what the likelihood of
things happening were

You could make
all of the predictions

we make today
and take for granted

You could make them
using mathematics

NARRATOR: It was
a fundamental insight,

and one of the doors that
led to the modern world

Inherent in all our attempts
to predict the future...

From the stock market

to insurance

to web retailers
trying to figure out

what you might buy next,

is the idea that
with the right data,

the likelihood of future
events can be calculated

[thunder roaring]

In fact, one of the
great success stories

in the science of prediction

yields a forecast that
many of us check every day

to answer the question,
"Do I need an umbrella

or a storm shelter?"

♪♪

The hurricane season of 2017

will be remembered for
its ferocity and destruction

The strongest ever on record

The Puerto Rico and the San
Juan that we knew yesterday

is no longer there

[helicopter hovering]

NARRATOR: The storms formed and
gained in intensity with surprising speed,

leaving forecasters to
emphasize the uncertainty

of where they might land

Maria is now a
Category Three hurricane

Exactly what it's going to
look like, we just don't know yet

There's still great uncertainty

NARRATOR: In
weather forecasting,

the only certainty
is uncertainty

LOUIS UCCELLINI:
One thing we know

for sure is we cannot
give you a perfect forecast

Given the nature of
how we make a forecast,

from the global observations to
equations running on computers,

stepping out in time,

I don't think there'll
ever be a perfect forecast

NARRATOR: To
fight that uncertainty,

forecasters have turned to
more data... lots more data

Here at the National
Weather Service

Baltimore-Washington office,

meteorologist Isha Renta

prepares for the afternoon
launch of a weather balloon

Twice a day, every day,

all across the U S
and around the world,

at the very same time,

balloons are released to
take a package of instruments

up through the atmosphere

It transmits readings about
every ten meters in height

ISHA RENTA: It's my understanding
that they have developed

other ways to get vertical
profiles of the atmosphere,

but still the accuracy
and the resolution

that the weather balloon
will give you is a lot higher

So that's why we
still depend on them

♪♪

NARRATOR: The data from
Isha's weather balloon ends up

at the National Center for
Environmental Prediction

in College Park, Maryland,

the starting point for
nearly all weather forecasts

in the United States

Her information becomes
one drop in a very large bucket

of data taken in each day

Temperature,
pressure, wind speed,

and direction in the atmosphere

Tens of thousands of
point observations are used

every hour of every day
as kind of a starting point

That's where we
begin the simulation,

from those observations

NARRATOR: It all
becomes part of a process,

which has been described

as one of the great
intellectual achievements

of the 20th century:
numerical forecasting

♪♪

The first step in
numerical forecasting

is to break a nearly 40-mile-
thick section of the atmosphere

into a three-dimensional grid

Then, each grid point is
assigned numerical values

for different aspects
of the weather,

based on the billions
of measurements

continually pouring
into the Weather Service

So you'll have an understanding

of temperature, pressure,

and values in terms of
wind and wind direction

at each one of these points

within this grid that
covers the globe

NARRATOR: From there,
equations from the physics

of fluids and thermodynamics
are applied to each grid point

CARBIN: Not only do you
change the characteristics

at each grid point, but the
changes at those grid points

affect neighboring grid points,

and then neighboring grid
points affect other grid points

And so you evolve the
atmosphere through time

in this three-dimensional space

NARRATOR: And
remarkably, the approach works

It's amazingly
crazy that it works

It's remarkable how
well it does work,

given that we're making
grand assumptions

about the initial
state, so to speak,

or the beginning
state of any forecast

NARRATOR: And that initial state
turns out to be absolutely crucial

♪♪

In the early days of
numerical forecasting,

it seemed like a definitive
weather prediction

extending far into the
future might soon be possible

But research in the 1960s

showed that slight errors
in measuring the initial state

grow larger over time,
leading predictions astray

So as you step ahead in time,

the forecast will
become less accurate

NARRATOR: Ironically, that
sensitivity to initial conditions

also suggested a way
to improve the accuracy

of numerical weather forecasts

Thanks to the power
of today's computers,

forecasters can run their
weather simulations not once

but several times

For each run, they slightly
alter the initial conditions

to reflect the inherent error
built into the measurements

and the uncertainty
in the model itself

The process is called
ensemble forecasting,

and the results are
called spaghetti plots

CARBIN: We're looking at
about 100 different forecasts here

for the jet stream
at about six days ago

We have the actual jet stream

drawn as the white
line on here today,

and you can see most of
the forecasts six days ago

were well north of where
we actually find the jet stream

this morning

And then we'll go
to a five-day forecast

and a four-day forecast
and a three-day forecast

and then down to two days
and the day of the event

And you can see how the
model forecasts all converge

on that solution, which is what
you would expect them to do

But you go back to
the six-day forecast,

you can see the large spread
in the ensemble solutions

for this particular pattern

NARRATOR: In the end,
meteorologists turn to statistical tools

to analyze weather forecasts

and often use probabilities
to express the uncertainty

in the results

That's the "40% chance
of rain" you might hear

from your local forecaster

Meteorology is
probabilistic at its very core,

and I believe that the
general public knows

there is uncertainty
inherent in everything we say,

but we're getting better

♪♪

Our forecasts for three
days out now are as accurate

as one-day forecasts
were about 10 years ago

And this continues to improve

So the science has advanced
beyond my wildest dreams,

and it's hard to even see
where it might go in the future

NARRATOR: Just
like in meteorology,

for the rest of science,

the ultimate test of
our understanding

is our ability to make
accurate predictions

On a grand scale,
scientific theories

like Einstein's general
theory of relativity

have to make predictions

that can be tested
to become accepted

In that case, it took four years

before a full solar
eclipse revealed

that light passing
near the sun curved,

just as predicted by
Einstein's theory...

The first proof he was right

that the sun's mass distorts
the fabric of space-time...

What we experience as gravity

[child babbling]

In fact, the scientific method
demands a hypothesis

which leads to a
prediction of results

from a carefully
designed experiment

that will test its claim

Surprisingly, it wasn't
until the 1920s and '30s

that a British scientist,
Ronald A Fisher,

laid out guidelines for
designing experiments

using statistics and probability
as a way of judging results

♪♪

As an example, he
told the story of a lady

who claimed to
taste the difference

between milk poured into her tea

and tea poured into her milk

Fisher considered
ways to test that

What if he presented her
with just one cup to identify?

If she got it right one
time, you'd probably,

"Well, yeah, but she had a


of getting it right"

So you'd be pretty
unconvinced that she has the skill

NARRATOR: Fisher proposed
that a reasonable test of her ability

would be eight cups,

four with milk into tea,

four with tea into milk,

each presented randomly

The lady then had to separate
them back into the two groups

Why eight?

Because that produced


of the cups, but only one
with them separated correctly

If she got it right,

that wouldn't prove
she had a special ability,

but Fisher could conclude,
if she was just guessing,

it was an extremely
unlikely result,

a probability of
just 1 4 percent

Thanks mainly to Fisher,

that idea became enshrined
in experimental science

as the "p-value"...
P for probability

If you assume your results
were just due to chance,

that what you were
testing had no effect,

what's the probability you
would see those results

or something even more rare?

If you assume that
there's a process

that is completely random,

and you find that it's pretty
unlikely to get your data,

then you might be suspicious
that something is happening

You might conclude, in fact,
that it's not a random process

That this is interesting to look
at what else might be going on,

and it passes some
kind of sniff test

NARRATOR: Fisher also
suggested a benchmark:

only experimental results

where the p-value
was under 05...

A probability of less
than five percent...

Were worth a second look

In other words,

if you assume your results
were just due to chance,

you'd see them less
than one time out of 20

Not very likely

He called those results
"statistically significant"

Statistically significant

Now this is a terrible word

It could be quite insignificant

You could be detecting a
very, very, very small effect,

but it would be called,
in the mathematical lingo,

"significant"

♪♪

NARRATOR: Since Fisher's day,

p-values have been used as a
convenient yardstick for success

by many, including
most scientific journals

Since they prefer
to publish successes,

and getting published is
critical to career advancement,

the temptation to massage and
manipulate experimental data

into a good p-value is enormous

There's even a name
for it: "p-hacking"

P-hacking is when researchers
consciously or unconsciously

guide their data analysis to
get the results that they want,

and since 05 is kind of the-the
bar for being able to publish

and call something real,
and get all your grant money,

it's usually guiding the results

so that you arrive
at that p of 05

♪♪

NARRATOR: How much p-hacking
really goes on is hard to know

What may be more
important is to remember

what was originally
intended by a p-value

The p-value was always meant
to be a detective, not a judge

If you do an experiment

and find the result that
is statistically significant,

that is telling you, that is
an interesting place to look

and research and understand
further what's going on,

not "don't study this anymore
because the matter is settled"

NARRATOR: In a sense,
a low p-value is an invitation

to reproduce the experiment,
to help validate the result,

but that doesn't always happen

In fact, there are few
career incentives for it

Journals and funders
prefer novel research

There is no Nobel
Prize for replication

Another solution to p-hacking
and the overemphasis on p-values

may simply be
greater transparency

WILLIAMS: More and
more, what people are doing

is publishing their data

And so it's becoming harder
and harder to lie with statistics,

because people will
just probe and say,

"Well, give me the
set you analyzed

and let me see how
you got this result"

NARRATOR: Statistics continues
to play a fundamental role in science,

but really anywhere
data is collected,

you'll find statisticians
are at work,

looking for patterns,
drawing conclusions,

and often making predictions...

Though they don't
always work out

The presidential
election of 2016

was a tough one for pollsters,

the folks who conduct
and analyze opinion polls

Hillary Clinton was the
overwhelming favorite

to beat Donald Trump
right up to election day

Trump is headed
for a historic defeat

He's going to
lose by a landslide

I think that she's going
to have a very good night

NARRATOR: The "New York
Times" put Trump's chances at 15%

One pollster on election
night gave him one percent

A projection of a 99% chance
of winning, is that correct?

The odds are overwhelming

of a Hillary Clinton
victory on Tuesday

I would be very surprised
if anything else happened

NARRATOR: And, of course,
Trump won, and Clinton lost

CHALABI: People
were repeatedly told,

"Hillary Clinton
is the candidate

most likely to win this
election," and she didn't

And I think that really left
people feeling almost lied to,

almost cheated by these numbers

NARRATOR: So what
was going on with the polls?

And exactly how do
people predict elections?

One way is just by asking
people who they'll vote for

PATRICK MURRAY: One of
the great things about polling

is that we don't have
to talk to everybody

in order to find out what
the opinions are of everybody

We can actually select
something called a sample

♪♪

NARRATOR: Sampling
is a familiar idea

To see if the soup is right,

you taste a teaspoon,
not the whole pot

To test your
blood at the doctor,

they typically draw
less than an ounce,

they don't drain you dry

But in many circumstances,

finding a representative
sample is harder than it sounds

WILLIAMS: Let's
suppose that this

is the population of about
a thousand people in a city,

and we want to know,
"Are people for or against

converting a park
into a dog park?"

And so these green
beads down here

are going to represent
people who were for it,

and the red beads are
folks who are against it

NARRATOR: Talithia's
first step is to take advantage

of an unlikely ally in
sampling: randomness

MURRAY: The
beauty of randomization

is that as long as you throw
everything from your population

into one pot and
randomly pull it out,

you can be sure
that you're within

a certain percentage points

of the actual value
that's in that pot

NARRATOR: So the plan is to randomly
sample the beads... but how many?

That depends on how
much accuracy Talithia wants

One measure is
the margin of error...

The maximum amount
the result from the sample

can be expected to differ from
that of the whole population

It's the plus or minus
figure, often a percentage,

you see in the
fine print in polls

But there's also
confidence level

Inherently, there is uncertainty

that any sample really
represents a whole population

The confidence level tells
you how sure you can be

about your result

A 90% confidence level means,

on average, if you ran your
poll or sample 100 times,


it would be accurate,

within the margin of error

Talithia knows the total
number of beads is a thousand

And she's settled on a
plus-or-minus five-percent

margin of error at a


That means she needs a
sample size of at least 214 beads

WILLIAMS: Here are the results:

We got a 103 red
beads and 111 green,

so about 48% of our
population would vote against,

and about 52% would vote for

Now, remember that margin
of error that we talked about,

that plus-or-minus five percent?

So once you take
that into account,

those numbers really
aren't that different at all

So I guess you could say,
this puppy is too close to call

NARRATOR: In fact, within the
margin of error, the stats got it right

There were an equal number
of red and green beads in the jar

While the sampling error
built in from the mathematics

can be quantified,

there are other
errors that can't

MURRAY: The other parts of the
error... how we word our questions,

how the respondents
feel that day,

the responsibility to predict

what their behavior is going to
be somewhere down the line...

All those sources of error

are something that
we can't calculate

NARRATOR: And there's a catch
to random sampling for polls too

A few decades back,

when just about every
household had a landline,

finding a random sample meant
randomly dialing phone numbers

MURRAY: Into the 1970s
and the 1980s, we were getting,

you know, 90% response rates

If we randomly chose
a phone number,

somebody on the other end
of that phone would pick it up

and would do the
interview with us

[phone rings]

NARRATOR: Those days are over

Thanks to caller I D
and answering machines,

people often don't answer
their landlines anymore...

If they even have one

Response rates are way down

NATE SILVER: Only
about ten percent of people

respond to polls

So you're kind of
crossing your fingers

and hoping the people you reach

are the same as the ones
that are actually going to vote

For example, we found in


enough white voters
without college degrees

RICHARD DE VEAUX: If there's a bias
in the data, you cannot recover from it

As we've seen from
some recent elections

[chuckles]

NARRATOR: After Donald
Trump's surprise win,

many wondered if
polling was broken

But if you look at
the polls themselves,

and not the headlines,

on average, polls on the
national and state level

were off by historically
typical amounts

SILVER: So when I hear people
say, "Oh, the polls were wrong,"

then it probably reflects
people's interpretations

about the polls being wrong,

where people, for
various reasons,

looked at the
polls, and they said,

"These numbers prove to
me that Clinton's going to win"

When we looked
at the polls, we said,

"These numbers certainly
make her a favorite,

"but they point toward an
election that's fairly close

and quite uncertain, actually"

NARRATOR: And in 2016,

the U S presidential
election was just that close

Trump's victory
depended on fewer votes

than the seating capacity of
some college football stadiums...

Spread across three states:

Pennsylvania,
Wisconsin, and Michigan

And there were some problems
with the polls in those states

that led to underestimating
Trump's support,

according to a postmortem
by a consortium of pollsters

♪♪

Nate Silver, the founder of
the website FiveThirtyEight,

is one of the biggest
names in polling...

Even though he doesn't
generally conduct polls

SILVER: Our job is to
take other people's polls

and to translate that

in terms of a probability,
to say basically whether...

Um, who's ahead,

which is usually
pretty easy to tell, um,

but then how certain or
uncertain is the election

is the more difficult part

NARRATOR: Like a meteorologist,

Nate presents his
predictions as probabilities

On the morning of
Election Day 2016,

he gave Clinton about
a 70% chance of winning

and Trump about a 30% chance

That's like rolling
a ten-sided die

with seven sides
that are Clinton

and three that are Trump

SILVER: People who
make probabilistic forecasts,

they're not saying that
politics is intrinsically random

They're saying that we have
imperfect knowledge of it,

and that if you think you
can be more certain than that,

you're probably fooling yourself
based on how accurate polls,

other types of
political data are

♪♪

NARRATOR: Ultimately,
interpreting a probability

depends on the situation

While a 30% chance
might seem slim,

if you learned the flight
you were about to board

crashed three out
of every ten trips,

would you get on the plane?

FLIGHT ATTENDANT: As this
plane only makes it to its destination

seven out of ten times,

please pay attention to
our short safety briefing

NARRATOR: Or if a
weather forecaster said

there's only a


and then it rained...

Would you care?

ELLENBERG: If it does rain,

no one demands to
know, "Why did it rain?

We have to get to
the bottom of this"

We can say like, "It just did"

It might have rained,
it might not have rained

As it happened, it did

I do think there's a
certain natural resistance

to seeing things that
maybe we care about

more than whether
it's going to rain or not,

like elections, in that same way

NARRATOR: As 2016 shows,

predicting who will
win the U S presidency,

a one-time contest between
two unique opponents,

is far from easy

But in at least one field,

there are literally decades
of detailed statistics

on how the
contests played out...

Baseball

♪♪

Baseball has always
been a game of numbers...

Box scores, batting
averages, ERAs, RBIs

But while stats have
always been part of baseball,

in the last 20 years, their
importance has skyrocketed

due to sports analytics,

the use of predictive models to
improve a team's performance

To some extent every
business, not just sports,

is really trying to predict
the next event, you know

Whether you're on Wall Street,

or if you're in the
tech business,

what's the new new thing

And for us, it's future
player performance

NARRATOR: Billy
Beane was one of the first

to adopt the quantitative
approach in the late '90s,

when he was the general
manager of the Oakland Athletics

Stuck with the low payroll
of a small-market team,

he abandoned decades
of subjective baseball lore

and committed the organization
to using statistical analyses

to guide the team's
decision-making

It very much became a
mathematical equation

putting together a baseball team

NARRATOR: Billy's stats-driven
approach started to attract attention

when the Oakland A's
finished in the playoffs

in four consecutive years

and set a league record
with 20 wins in a row

Then it was lionized

and even given a name in a
best-selling book and movie,

"Moneyball"

Brad Pitt plays Billy

BRAD PITT: If we win on
our budget with this team,

we'll have changed the game

♪♪

NARRATOR: While
"Moneyballing" didn't lead

to a league championship
for the Oakland A's,

it did change the game

Today, every Major
League Baseball team

has a sports
analytics department,

trying to predict and enhance
future player performance

through data,

analyzing everything

from the angle and speed of
the ball coming off the bat...

To which players
should be brought up

from the minor leagues or traded

BEANE: I'll never
pretend to be a math whiz,

I just understand its
powers and its application

When you run a Major
League Baseball team,

which is a great job,

and every kid who
dreams of doing it,

I can tell you it's
everything you've thought of

But when they ask me,

"What do I have
to do to do that?"

My answer is always the same

I say, "Go study
and get an A in math"

NARRATOR: While sports
analytics has transformed baseball,

Moneyballing has found its
way into many unrelated fields

Proponents of data-driven
decision making and prediction

have applied the approach
to areas as diverse

as popular music
and law enforcement

Moneyballing has been enabled

by the vast amounts
of information

gathered through the
internet, so-called "Big Data"

Our current output of data

is roughly 2 5
quintillion bytes a day

[splashing]

But what about the
opposite situation,

when there's very little data,
yet actions need to be taken...

For example when searching
for people lost at sea?

How do you even begin to
predict where they might be?

The U S Coast Guard's
Sector Boston Command Center

From this secure set of rooms,

the Coast Guard coordinates
all operations in the Boston area,

including national
security, drug enforcement,

and search and rescue

[phone ringing]

Good morning, Coast Guard
Sector Boston Command Center,

Mr. Fleming speaking

PHONE CALLER:
Uh, good morning, sir

NARRATOR: A caller reports

that a friend went
paddleboarding

earlier in the morning,
but he's now overdue

[alarm blaring]

The Coast Guard
initiates a search

with a 45-foot response boat

EDWARD NYGREN: Engaging

NARRATOR: out of Boston Harbor

NYGREN: Coming up

NARRATOR: Unfortunately,
a paddle craft in trouble

has grown increasingly common

NYGREN: You are required
to have a life jacket on

The reason for that is in 2015,

I think we had 625
deaths nationwide...

A number of those
people that were recovered

were recovered
without a life jacket

NARRATOR: The Command
Center also launches another boat

out of Station Point
Allerton, in Hull

CREWMAN: Short tack disconnected

LUKE SCHAFFER:
Stand clear of lines

[horn blowing]

NARRATOR: The caller said the
missing person typically paddled

between Nantasket
Beach and Boston Light,

about three miles away

But with all the unknowns...

Where he got into trouble
and how he may have drifted...

The search area could be
as large as 20 square miles

Search and rescue operations

are often based on
unique circumstances

and require action, despite
incomplete information

To attack problems like that,

statisticians turn to
an idea that originates

with an 18th-century
English clergyman

interested in probability...
Thomas Bayes

Imagine you are
given a coin to flip,

and you want to know if it
is fair, 50-50 heads or tails,

or weighted to land
more on heads than tails

The traditional approach
in statistics and science

doesn't assume either answer
and uses experiments to find out

In this case that involves
flipping the coin a lot

Or you could approach
the problem like a Bayesian

Unlike traditional statistics,

that means starting
with an initial probability

based on what you know

In this case, all the coins
you've ever come across

in a lifetime of flipping
coins have been fair

It seems likely this
one is probably fair too

Next, you also flip the coin,

updating the
probability as you go

Let's say it starts off with
several heads in a row

That might make you wonder,

increasing your probability
estimate that it's weighted

But as you flip it more times,
those start to look like chance

In the end, your best estimate

is that it is
probably a fair coin,

but you are open to
any new information

Like it belongs to your uncle
the con man, "Crooked Larry"

NYGREN: Sector 659

Our estimated time of
arrival is one-one-five-eight

NARRATOR: Bayesian inference
creates a rigorous mathematical approach

to calculating probabilities
based on new information

And it sits at the heart
of the Coast Guard's

Search and Rescue Optimal
Planning System: SAROPS

BRIAN FLEMING: He's been
missing since 7:30 this morning,

so I'm going to go ahead
and do a SAROPS drift

NARRATOR: SAROPS
takes information

about the last-known position
of the object of the search

FLEMING: What's the
direction of the wind?

NARRATOR: along with the
readings of currents and winds

and combines
them with information

about how objects
drift in the water

to simulate thousands
of possible paths

the target may have taken

These get processed
into probabilities,

indicated by color,

and turned into search
plans to be ex*cuted

SCHAFFER: SAROPS is really
a workhorse for the Coast Guard

It does a lot of the
calculations for us

It provides us with a lot
of valuable search patterns

and search-planning options

MAN [on phone]: I thought he was
pretty far off shore but, you know,

he said he was
okay, so I kept going

NARRATOR: Word of
the search has spread

A boater calls in a sighting
from earlier in the day

What I did is I went in and put
that information into SAROPS,

and it changed everything

NARRATOR: SAROPS quickly
recalculates all the probabilities

and generates a new search plan

The area has shifted about
three miles farther out to sea

NYGREN: We are on-scene,
commencing search pattern now

♪♪

Keep a good look out

CREWMAN: Roger, coming up

We're assessing
the situation on scene

♪♪

SCHAFFER: Any object you see in
the water, please take a closer look at

♪♪

Paddleboarder, port side

♪♪

CREWMAN [on radio]: Roger,
we have located a paddleboarder

with zero-one person on board

Off the port corridor!

CREWMAN: Starboard side

I have a visual

CREWMAN: All right

NARRATOR: As it turns
out, the search has been a drill

Hours earlier, the paddleboard
was placed in the water

by another Coast Guard
ship and allowed to drift

The instruments mounted
on it are there to measure wind

and record the path it's taken,

information that
will later be used

to tweak the drift
simulations in SAROPS,

though the system
performed quite well today

SCHAFFER: The object was right
in the middle of our search patterns

So SAROPS was
actually dead-on accurate

in predicting where
we needed to search

to find the missing
paddleboarder

♪♪

FLEMING: To be able
to call a family and say,

"Your family and
friends is coming home,"

is absolutely a call

that all of us should
have the chance to make,

and, fortunately,
because of stuff like this,

we do get to make that call

♪♪

NARRATOR: The computational
complexity of updating probabilities

held the Bayesian approach
back for most of the 20th century

But today's computing power
has unleashed it on the world

It's in everything
from your spam filter

to the way Google searches
work to self-driving cars

Some even find in the
Bayesian embrace of probability,

similarities to how we
learn from experience

And they've built
it into computers,

Making it part of a
powerful new force:

machine learning

SEBASTIAN THRUN: In the past,
when we programmed computers,

we tended to really write
down, in excruciating detail,

a set of rules that
would tell the computer

what to do in every
single contingencies

NARRATOR: But there's
another approach...

To treat the computer

like a child learning
to ride a bike

No one teaches a child
to ride using a set of rules

There may be some tips,

but ultimately, it
is trial and error...

Experience...
That's the instructor

THRUN: The new thing,
the new kid on the block

is machine learning,

specifically something
called deep learning

Here, we don't inform
the computer of the rules,

but through examples

So similar to,
like, a small child

that falls down and
learns from this experience,

we just let the computer
learn from examples

NARRATOR: Suppose
you want to train a computer

to recognize pictures of cats

By scanning through
thousands of labeled pictures...

Some cats, some not...

The computer can
develop its own guidelines

for assessing the probability
that a picture is a cat

And these days

computers are doing far
more than just looking for cats

THRUN: Some of the
best computers now

can learn how to beat the
world's best Go champion

or to discover documents
in stacks of documents,

work that highly paid
lawyers normally do,

or diagnose diseases

At Stanford, we
recently ran a study

to understand whether a
machine-learning algorithm

can compete with
top-notch, Stanford-level,

board-certified dermatologists

in spotting things
like skin cancer

And Io and behold, we found
that our machine-learning algorithm,

our little box,

is as good as the best human
doctor in finding skin cancer

NARRATOR: That
raises a lot of questions:

should we trust software
over our doctors?

Or are diagnostic
programs like Sebastian's

the intelligent medical
assistants of tomorrow,

a new tool but not a substitute?

And there are other concerns

If you asked a person riding
a bike exactly how they do it,

they'd be hard-pressed
to put it into words

The same is true

with so-called "black box"
machine learning applications

like Sebastian's:

no one, including Sebastian,

knows how it detects skin cancer

Like the bicyclist,
it just does,

which may be fine
for diagnostic software,

but not for other
aspects of medicine,

like treatment decisions

DANIELA WITTEN: If
what you're doing is deciding

what dose of chemotherapy
to give a patient,

I think most people
would be uncomfortable

with that being a black box

People would want to understand

where those predictions
are coming from

NARRATOR: The same can be true

for evaluating who
should get a home loan,

or who should get fired from
their job for poor performance,

or who gets paroled,

all situations

in which black box machine
learning software are in use

WITTEN: These are algorithms

that can have a big
effect on people's lives

And we have to
understand, as a society,

what is going into
those algorithms

and what they're based on,

in order to make sure
that they're not perpetuating

social problems
that we already have

♪♪

NARRATOR: We live in an age
when the fusion of data, computers,

probability, and statistics

grants us more predictive power
than we've ever known before

We can see the
tangible benefits,

and some of the dangers,

while also wondering
where this will all go

We're really seeing a
new science of statistics

developing under our feet

That's exciting,

and I think it must
be a little bit like

what it was like when
the theory of probability

was first being developed

by Pascal and Fermat
and people around them,

that people were sort of saying,

"My God, these are
questions that mathematics

can really have
something to say about"

I think that must have
been what it was like

when statistics in
its traditional form

was being developed in the
first part of the 20th century,

and suddenly people
were just asking

whole new kinds of questions

that they couldn't even
have approached before

And I think we're having
another moment like that now

♪♪

NARRATOR: While tomorrow
will always remain uncertain,

mathematics will
continue to guide the way,

through the power
of probability,

and prediction by the numbers