NARRATOR: The stage is set for a
showdown the world will be watching
All eyes will be
on one competitor,
but he's not breaking a sweat
We're not talking
nerves of steel
More like a rock-solid
memory of silicon
WATSON: My name is Watson
NARRATOR: For
the first time in history,
a computer will compete
in the ultimate brain game
ANNOUNCER: This is Jeopardy!
NARRATOR: For the
scientists who created Watson,
the excitement and
tension are palpable
It's going to be
edge of your seat,
It's going to be nerve wracking
NARRATOR: A four-year
odyssey comes down to this
DAVID FERRUCCI: What really is
going to happen and you just don't know
You don't know
NARRATOR: There are
more than reputations at stake
Some say a Watson victory
could signal a new age,
a revolution in
artificial intelligence
LUIS VON AHN: It used to be the
case that intelligence was chess, right?
If you can play chess,
that's intelligence
NARRATOR: But taking on spoken
language and ferocious wordplay
means engaging by a
whole new set of rules
It requires more
than just memory
to match wits with humans
Watson?
Just understanding the
question is a pretty big deal
NARRATOR: How will Watson do it?
NOVA gets unique access
to the making of Watson...
Shadowing the IBM team
working around the clock,
training and test driving Watson
MAN: comedies
directed by Blake Edwards
Watson?
WATSON: What is
The Pink Panther?
And we couldn't write
rules for every combination
of word and phrases and context
NARRATOR: Refining
powerful new tools of computing
FERRUCCI: We took a huge
jump with machine learning
We were like, "Whoo!"
TODD CRAIN: Watson?
Yes, he got it!
NARRATOR: And creating a
machine that can learn from its mistakes
WATSON: What is mosquito?
No
WATSON: Holiday?
No
WATSON: Artificial sweetener?
CRAIN: Come on now, no
We don't understand the question
We don't understand the category
That was one of the
tensest days I've ever had
FERRUCCI: You
fall flat on your face
Did I expect to get fired?
No, but maybe
We came back and the
game was neck and neck
Wordsworth said they
soar, but never roam
Watson?
WATSON: What is skylark?
That is correct
NARRATOR: Is Watson ready to
face the game's all-time champions?
This is too daunting
a task for a computer
HARRY FRIEDMAN:
I think we've gone
from impressed to blown away
NARRATOR: When man
takes on machine, who will win?
CRAIN: Watson?
WATSON: What is Crete?
CRAIN: Yes
NARRATOR: NOVA takes you
inside the world of artificial intelligence
and the quest to build a
computer like none before it
Good luck, Watson
WATSON: "Smartest Machine
on Earth"... right now, on NOVA.
Major funding for NOVA
is provided by the following
And
And by the Corporation
for Public Broadcasting
And by contributions to
your PBS station from:
MAN: We're ten seconds
away on tape two.
NARRATOR: Dave
Ferrucci is a nervous parent
He's spent the last
four years building
a revolutionary new computer,
and it's about to
face its biggest test
in front of an
audience of millions
WATSON: Hello, my name is Watson
I hope we will have
a good game today,
but first I have
to test my voice
NARRATOR: When the cameras roll,
the computer, called
Watson, will make history
as it competes on the
popular quiz show Jeopardy!
ANNOUNCER: This is Jeopardy!
It's frightening, right?
It's a different experience
It's a very different
experience for a scientist
to sit here and
have this happen live
MAN: Six, five, four
NARRATOR: Ferrucci
has reason to fear
Watson is playing for
a million-dollar jackpot
against the game's
toughest competitors:
Brad Rutter, Jeopardy!'s
biggest money winner,
and Ken Jennings, famous for
winning 74 consecutive games
KEN JENNINGS: There's
some contestant's pride
I want to beat my human
competition but, you know,
as a species I would
like mankind to beat
the big, bad computer
WATSON: Let's finish
"Leaders of World w*r II"
NARRATOR: This big, bad
computer is the culmination
of four years' intensive work
IBM has put Watson through
hundreds of practice games,
with a stand-in host
and real contestants
CRAIN: After Germany
invaded the Netherlands,
this queen, her family,
and cabinet fled to London
Who is Beatrix?
CRAIN: No Watson?
WATSON: Who is Wilhelmina?
CRAIN: That is correct
ERIC BROWN: It's a human standing
there with their carbon and water,
versus the computer
with all of its silicon,
and its main memory and its disc
NARRATOR: It seems like it should
be easy for the computer to win,
with its enormous memory
and processing power
But the human brain makes
an intimidating opponent,
especially on Jeopardy!
Jeopardy! questions are tricky
They have puns in them,
they have little jokes in them
Just understanding the
question is a pretty big deal
CRAIN: This trusted friend
was the first nondairy
powdered creamer
Watson?
WATSON: What is milk?
[laughter]
CRAIN: No! Maria?
What is Coffee-mate?
CRAIN: Thank you
FERRUCCI: Humans
communicate very fluently
in, you know, natural language,
and that's where computers
struggle dramatically
NARRATOR: Now, Watson is on
the verge of conquering that challenge
CRAIN: Here we go
A garment worn by a child,
perhaps aboard an operatic ship
Watson?
WATSON: What is pinafore?
CRAIN: Yes How did you get that?
NARRATOR: If Watson wins on
Jeopardy!, it will be a major breakthrough
in a quest that's
gone on for decades:
the audacious dream to build a
machine as smart as a person;
the quest for artificial
intelligence... AI
I am Elektro
My brain is bigger than yours
When we started doing AI,
the goal was, why
can't we build a person?
We all know how to
make people, that's easy
What if we could
build one out of silicon?
NARRATOR: Pioneers
drew their inspiration
from the world
of science fiction
As a child, Isaac
Asimov turns up
So here I'm an
adolescent and the robots,
intelligent machines,
are a part of my life
At your service
Let's see about
getting them built
NARRATOR: In the early days,
computers grew rapidly more powerful,
quickly mastering
complex equations
The first programs
we wrote at M I T
solved problems that only very
educated people could solve,
like problems in
calculus and then algebra
NARRATOR: The computer pioneers
thought they were on a fast track
to building
humanlike intelligence
I confidently expect
that within 10 or 15 years,
we will find emerging
from the laboratories
something not too far from
the robot of science fiction fame
PATRICK WINSTON: In
the beginning we thought,
"Well, maybe ten or 15
years and we'll have something
that's really smart"
WINOGRAD: In the beginning,
people really were amazed
at how much computers could do
When you see something
that's improving very fast,
you simply assume it will
continue improving that fast
indefinitely
NARRATOR: In the '60s,
confidence was so high
it inspired one of the
most iconic film characters
of all time
MAN: Hello, HAL, do you read me?
Do you read me, HAL?
HAL: Affirmative,
Dave I read you
When I was a kid, I saw
2001: A Space Odyssey
and HAL was just
the best thing ever
DAVE: Open the pod
bay doors, please, HAL
HAL: I'm sorry, Dave,
I'm afraid I can't do that
You know, it was a
murdering psychopath
HAL?
HAL?
But, it was intelligent,
could talk to people,
could see people,
could lip read,
could do all this stuff
A machine that could do that...
And I'd never even seen
a real computer at that time
HAL: Bishop takes knight's pawn
NARRATOR: More than 40
years after the creation of HAL,
no real computer or robot
has been able to interact
with humans as seamlessly
as Hollywood imagines
Tell me, what is love?
NARRATOR: The problem is our
own human computer, the brain,
a complex entity that's defied
any attempts at replication
We just had no idea how
sophisticated the brain was
NARRATOR: The computer
has always been king
when it comes to calculation
and processing
huge amounts of data
Pen, as in "the pen,"
what's the middle letter?
KIDS: "E"!
NARRATOR: But simple skills
that humans master early in life,
like understanding language
or recognizing objects,
continue to baffle researchers
You know, people
vastly misjudged
how subtle we are
when we're intelligent
People just hugely
underestimated that
NARRATOR: But the dream of
building a computer that could talk
and match wits with
humans never really died,
and a few years ago a
new plan was hatched,
sparked by an unlikely event
ANNOUNCER: This is Jeopardy!
ALEX TREBEK: In Wagner's operas,
this eldest Valkyrie is
stereotypically dressed
in a horned helmet
and breastplate
Ken?
Who is Brunhilde?
NARRATOR: In 2004, Ken
Jennings' 74 game-winning streak
on Jeopardy! Set
the country abuzz
and caught the eye of an IBM
executive while out to dinner
All of a sudden the entire
restaurant cleared out
to the bar that I'm sitting
at to go see Ken Jennings
TREBEK: This "Grand Ole Opry"
comedy star used to wear a straw hat
with the $1 98 price
tag still attached
Ken?
Who is Minnie Pearl?
Minnie Pearl, howdy!
NARRATOR: Charles Lickel
wondered if a computer could ever play
as well as Ken Jennings
So he pitched the idea to
some of IBM's top scientists
For the ones that
knew Jeopardy!,
they said, "Charles,
that's just too hard"
I think the prevailing view was
these questions were
difficult to understand,
difficult to even comprehend
what was being asked
Yeah, I was like, "No
way!" I was like, "No way!"
NARRATOR: But one researcher,
Dave Ferrucci, was intrigued
My view was maybe this
isn't as completely impossible
as we think it is
NARRATOR: For over 40 years,
Jeopardy! has been
pop culture's IQ test
Clues are given as answers
And contestants have to
respond in the form of a question
"Mothers," $1,600
TREBEK: It's a larger vessel
that guards and
supplies smaller ones
Christina
What is the mothership?
Mothership, yes
JENNINGS: The
show's central conceit
is a little syntactic reversal,
whereby they give you an
answer and you supply a question
You don't say
"George Washington";
you say, "Who is
George Washington?"
NARRATOR: To win, contestants
need to be human encyclopedias
It's essentially
everything under the sun
You know the categories
at the same second
Alex tells the folks
at home the category
One, you have to have
a broad knowledge,
because we have 13
categories on each show
Kate, start
"Ph" for $400
TREBEK: For the record,
Thomas Edison invented
the first practical
one of these in 1877
NARRATOR: Contestants
also have to be fast
What is the phonograph?
Good!
NARRATOR: They typically
have three seconds or less
to come up with an answer
TREBEK: The mortar and pestle
is a symbol of this profession
Ariel
What is a pharmacist?
Pharmacist is right
NARRATOR: To
compete on Jeopardy!,
IBM's computer must have
an enormous knowledge base
because it will not be
connected to the Internet
But the far bigger
challenge for the machine
will be understanding clues,
which can be extremely
convoluted or obscure
TREBEK: You'll find this
flower before "Pickle Bottom"
in a line of handbags
and bedding
And that would be petunia
Back to you, Ariel
JENNINGS: There'll
be a lot of puns
There'll be double meanings
And these are things
that computers, historically,
are terrible at
NARRATOR: Human language
is a minefield for computers
Consider this sentence
How's it go?
I shot an elephant
wearing my pajamas
Was I wearing the pajamas?
Was the elephant
wearing the pajamas?
So there are different
interpretations,
different ways to
parse the sentence
The word "shot"... What's
really going on there?
There's already
ambiguity in there
Could actually be sh**ting,
sort of a hunting sh**ting
If I'm a photographer and
I'm immersed in that context?
I may interpret that as
sh**ting with a camera
Which one did I mean?
You have to look at the context
NARRATOR: But a
computer has no context
It's just an electronic
brain in a box
In 2006, Ferrucci
tackles this challenge,
along with the best and
brightest programmers from IBM
and the country's top AI labs
To start, they run a test
GONDEK: We had an existing
state-of-the-art system
that people had worked on
for a number of years,
and we tried applying that
to the Jeopardy! challenge
NARRATOR: They feed one of IBM's
most sophisticated computer programs
hundreds of Jeopardy!
questions like this one:
The correct answer is
"Who is Edmund Halley?"
The computer says,
"Who is Peter Sellers?"
The computer ran a search
through a million documents,
looking for key
words from the clue
It homed in on a description
of one of the Pink Panther films,
in which one character
was a "paramour" or mistress
The star of the movie?
Peter Sellers
It's probably the last answer
a human would come up with
But it's typical for computers
The team has a long way to go
Just how far becomes clear
when they compare the computer
to the best human players
They create a graph
called "the cloud"
Each dot represents a Jeopardy!
Champion's performance
Jennings is at the top
What you see is a
cloud and it's around
they answer around
and they get around 90% correct
NARRATOR: And where is
the computer in this cloud?
GONDEK: If you ask it to
answer all the questions,
it would be giving you
You can't go on
Jeopardy! Like that
I mean, the best
humans are 90%, 92%
We weren't even close
NARRATOR: To win at Jeopardy!,
the team will need a whole new way
to tackle human language,
one that takes advantage of
the computer's basic strengths
At its electronic core,
a computer speaks a
very simple language:
binary code, on or off
But with that simple code
it can follow instructions
and solve complex problems
once reserved for
intellectual giants
It used to be the case that
intelligence was chess, right?
If you can play chess,
that's intelligence
NARRATOR: Computers
have mastered the game
Chess is easy for computers
because the rules are very
well defined and very clear
NARRATOR: The rules of
chess are relatively simple
A board of 64 squares
Each piece... pawn, knight,
queen... can move a certain way,
and there's a single goal:
take out your opponent's king
For humans, it is the
ultimate game of strategy
The way computers play chess
is not at all the way
people play chess
We humans look at the board
and have conceptual ideas like
control the center,
attack on the right
Very different from the
way computers play chess
NARRATOR: A
chess-playing computer looks at
virtually every possible move it
could make and every response,
every way the
game could play out
Computers play chess through
searching a tree of moves
down to a very deep level,
looking ahead on
every possible path
But they do it by brute force...
By going 20, 30, 40 moves ahead
and seeing all the bad
things that can happen
A person can't look that
many moves ahead, broadly
NARRATOR: This is the power
behind the most famous chess game
in the history of AI,
when in 1997, another IBM
computer named Deep Blue
beat the reigning world
champion, Gary Kasparov
REPORTER [archival]: The chess world
champion walked away from the match,
never looking back at the
computer that just beat him
NARRATOR: The victory
makes Deep Blue look pretty smart
But is it?
WINSTON: Deep Blue
it's only acting
as if it's intelligent
It's not really intelligent in
the way that we humans are
BROOKS: It's good at one
thing... it's playing chess
It can't do anything else;
it has no other
understanding of the world
It's just about chess moves
NARRATOR: This
lack of understanding
has hampered every
computer program
that's tackled human language
A perfect example is a program
from the 1960s called ELIZA
ELIZA was one of
the first programs
that had anything resembling
human conversation
It was a dialogue...
You typed things in,
it typed things back
FEMALE VOICE: How do you do?
Please tell me your problem
I'm feeling sad
Then it types back,
"Did you come to me
because you're feeling sad?"
NARRATOR: ELIZA was programmed
to respond like a psychiatrist
But it had no real insight
Instead, it followed
simple rules
and rearranged key phrases
So if I say "I'm dead,"
it responds, "Do you
enjoy being dead?"
It doesn't have
any understanding
that dead is a different
kind of condition
It really is just doing this
sort of fill-in-the-blanks
kind of pattern matching
NARRATOR: Anyone who tried
to solve the language problem
hit the same brick wall...
The computer's
profound ignorance
of what we take for
granted every day
There is just so much
more that we know
that we don't know we know
I mean, just we
know all kinds of stuff,
like you press the up
button in the elevator,
that means it's going to go up
Or, milk is white,
or water is wet
I mean, there's just
stuff that we know
that we don't even
realize we know
That's one of the
things that makes it hard
NARRATOR: All the commonsense
knowledge a human brain collects naturally
seems much too complex
to program into a computer
But that hasn't stopped
one scientist from trying
DOUG LENAT: So we have actually
manually entered about six million rules
That's about 3% of what
it's going to need to know
in terms of actually spanning
what you and I would call
"human common sense"
NARRATOR: For the last 25 years,
Doug Lenat has
been leading a team
trying to create
humanlike intelligence
by teaching a computer
common sense, rule by rule
The program is called
CYC, and at headquarters
the walls are covered
with logic diagrams
In a way, the magic of
this, the power of this,
is if you just tell it each
rule one by one by one
and you give it general
logical reasoning capabilities,
that's all you need to do
NARRATOR: So far,
CYC has six million rules
and can answer a lot of
commonsense questions
Like this one:
LENAT: Can a can cancan?
[cancan music playing]
NARRATOR: CYC says
no and it explains why
Here it's essentially
saying the reason
why cans can't cancan is
that cans are inanimate objects,
and it knows that
cancan dancing requires
at least partially having
a brain and using it
[cancan music continues]
It's just enough for CYC
to get the right answer
for the right reason
[music ends, applause]
NARRATOR: 25 years ago,
many experts considered
rules and logic the best hope
for building
artificial intelligence
But it's become clear
these alone are not enough
It's not just a matter of
piling in more and more stuff
There are basic principles
that we didn't understand;
putting in more and more stuff
doesn't get you basic principles
NARRATOR: At IBM, as Dave
Ferrucci and his team tackle
the Jeopardy! challenge,
they know that facts and
rules are just the beginning
We couldn't write rules
for every combination of
word and phrases and context
NARRATOR: They need a new
system, more fluid and flexible,
to navigate the twists and turns
of many different kinds
of Jeopardy! questions
They name their system Watson
after IBM founder Thomas Watson
The electronic Watson
consists of 2,800 processors
That's like 6,000
high-end home computers
All together, it's the
size of 10 refrigerators
The team starts filling
his memory banks
with about 10 million documents,
most downloaded
from the Internet
Because when
Watson plays Jeopardy!,
he must stand alone, just
like his human competitors
FERRUCCI: All kinds
of content, okay...
Encyclopedias, dictionaries,
thesauri, books, plays,
you name it
NARRATOR: The entire World
Book Encyclopedia, Wikipedia,
the Internet Movie Database,
much of the New York
Times archive and the Bible
are just some of
Watson's resources
And to build on Watson's
foundation of data and rules,
the team turns to a powerful
tool in the computing world
It's called "machine learning"
Machine learning is just like
human learning from examples
VON AHN: Before people would
just write rules, write rules by hand
Nowadays, it's all
based on examples
NARRATOR: To understand
how machine learning works,
consider for a
moment the letter "A"
What if you had to
describe it to a computer?
It's a real problem faced
by the U S Postal Service,
whose computers must
decipher all kinds of addresses,
printed and handwritten
VON AHN: We all know
what an "A" looks like
I know when I see it,
but there's just way too
many different types of "A"s
There are fonts where the
"A" is just a triangle pointing up,
that's an "A"
Pretty quickly, you realize,
there is no simple set of rules
that you can write
down currently
for a program to determine
whether a letter
is an "A" or not
NARRATOR: Humans might not
be able to come up with the rules
that reliably identify
all kinds of "A" s,
but it turns out a
computer can do it for itself
if you give it enough examples
The way you do it
is you just get an "A",
send it to the program
and say that's an "A"
Here is another "A",
different one, that's an "A"
Here's another "A", it's a
different one, that's an "A"
Then you would give
it another example
and you would give
it another example
and you would do
that a million times
NARRATOR: The computer hunts
for patterns among all those examples
and it finds them
So the next time it
meets a letter "A"...
Even one it hasn't seen
before... It will recognize it
This is machine learning,
and it's a crucial element
of Watson's programming
The team trains Watson,
but here, instead of letters,
the examples are
tens of thousands
of old Jeopardy! questions
along with a cheat sheet
of all the correct answers
Using machine learning,
Watson will hunt for patterns
between the type of
question, the correct answer
and the kinds of evidence
that support that answer
Now we do this over
thousands of questions,
so we come up with some way
to weigh the evidence on average,
so that we come up
with the right answer
NARRATOR: Now, when he's
faced with a brand new question,
Watson uses what he
learned from these patterns,
and declares his confidence
in each possible answer
FERRUCCI: In the end
we get a list that says,
"Here's the top answer,
and we're 75% sure it's right"
NARRATOR: Watson has now
become a complex architecture
of rules, raw data
and machine learning
that enables him
to use statistics
to choose the right answer
To test out this system,
the team scours the
halls for IBM employees
who can play Jeopardy!
And everyone squeezes
into a conference room
HOST: "In 1978, New Jersey
Monthly reporter Steven Levy
famously found this man's brain"
Watson?
WATSON: What is Einstein's?
HOST: Albert Einstein's, yes
[laughter]
HOST: "The Fifth Amendment says
"that private property shall
not be taken for public use
without this"
Watson?
WATSON: What is
"just compensation"?
HOST: Yes
NARRATOR: With this new system,
Watson surges into the winners' cloud
We took a huge jump
with machine learning
HOST: Watson with a
commanding lead, $24,863
We saw a huge
jump in performance
We were, like, "Whoo!"
NARRATOR: Up to now,
appearing on the TV
show has only been a dream
But Watson is performing
so well, Dave Ferrucci decides
it's time to call Jeopardy!
In December 2009, Jeopardy!
Producers arrive at IBM
to size up Dave
Ferrucci's new creation
Like any human contestant,
Watson must audition
to earn a spot on the show
FERRUCCI: We spent all this time,
you know, developing this system
and pushing its capabilities,
then here you are sitting
here, all the executives are there
[laughter]
HARRY FRIEDMAN: You
hear "computer," you think,
well, of course a computer
should have all the answers
You hear about Q&A technology
Well, isn't this just
a big search engine?
FERRUCCI: And
they're waiting to see,
you know, what really
is going to happen
And you just don't
know, you don't know
NARRATOR: To impress the
executives, IBM builds a makeshift studio,
hires comedian Todd Crain
to act as game show host,
and brings in former
TV contestants
GONDEK: That was one of
the tensest days I've ever had
Because we had never seen it
play against Jeopardy! players
CRAIN: Select again, David
GONDEK: And I remember, like,
the day before, we're tuning everything
I was putting in the
best strategy that we had
I was putting the best stuff
that we had and I thought,
"Well, this is just
gonna k*ll 'em!"
CRAIN: Miranda?
What is "The Cat's
in the Cradle"?
That is correct
What is "I am the Walrus"?
Yes!
MIRANDA: What
is "Crocodile Rock"?
Yes
GONDEK: They were just
like professional athletes
It was a really tough
few games for us
NARRATOR: In the first round, it
seems that Watson is auditioning
not for a game
show, but a sit-com
Where do we go next?
WATSON: L, underscore,
underscore, underscore, underscore, O
Whoa for 1,000
NARRATOR: There are suddenly
unexpected bugs that need fixing
We weren't dealing with
Roman numerals well
So it'll say like "Henry the
CRAIN [reading]:
along with his weak-minded
half-brother, Ivan the Fifth"
CRAIN: Watson?
WATSON: What is "Peter"?
CRAIN: More specific?
WATSON: What is "Peter 'Eye""?
[laughter]
No
Carrie or David?
Carrie? Who is Peter the Great?
That is correct
NARRATOR: In Final Jeopardy!,
where contestants must place bets
and write down the
answer, things only get worse
Under the category
"Flags," the clue is
NARRATOR [reading]:
[Final Jeopardy
theme music playing]
You need to know
a little bit something
about 18th-century flags
David, let's see if you did
What is the four-word motto
we're looking for, David?
What is, "Don't tread on me"?
That is correct!
Let's see if Watson got it right
[laughter]
NARRATOR: Watson
didn't recognize
the word "motto,"
and after scanning through
millions of documents,
he found the word "terrorism"
associated with "September 11"
so frequently, that
seemed like the best answer
By the time they
break for lunch,
it's humans, two; Watson, zero
And it's not clear if
Watson will ever be ready
for primetime
This was taking a risk for me
in the sense that you're sitting
here and saying, "You know what,
I think this is possible," and
then you fall flat on your face
and people say,
"Well, we're never going
to believe Ferrucci again"
Did I expect to get fired?
No, but maybe
[chuckles]
NARRATOR: But after lunch,
the producers are treated
to a different side of Watson
We came back, and the
third game was neck and neck,
incredibly competitive
"In Act Three of
an 1846 Verdi opera
[continues reading]:
Watson?
WATSON: What is Attila?
Be more specific?
WATSON: What is Attila the Hun?
Thank you very much
Attila the Hun... I'll take that
NARRATOR: That afternoon,
Watson climbs back in the game
"Wordsworth said they
soar, but never roam"
Watson?
WATSON: What is skylark?
CRAIN: That is correct
It's a device clamped to
the wheel of a parked car
with overdue tickets
Watson?
WATSON: What is boot?
CRAIN: Be more specific?
WATSON: What is Denver boot?
CRAIN: That is correct
This African-American
folklore laborer:
[continues reading]:
Watson?
WATSON: What is John Henry?
That is correct
Select again, Watson
NARRATOR: It may appear that
Watson has redeemed himself,
but the producers are troubled
by his erratic performance
Their verdict: Watson isn't
strong enough for Jeopardy!
At least not yet
[beeping]
Why is Watson so erratic?
To understand his weaknesses,
you have to appreciate
the complexity of the task
Consider this clue
The correct response
is "What is The Matrix?"
But how can Watson
figure that out?
First, he breaks down the
clue into grammatical parts,
identifying key
words and phrases
Then, Watson's powerful
search engines churn
through millions of documents,
including the Internet
Movie Database
What we do next is we
take these documents
and we pull out
candidate answers
And we'll pull out,
okay, Keanu Reeves
That could be a candidate
We'll pull out Nokia,
we'll pull out The Matrix.
NARRATOR: Other movies
starring Keanu Reeves
also become possible answers
GONDEK: We'll pull out The
Matrix 2, we'll pull out Speed,
Bill and Ted's Excellent
Adventure, all this stuff
Whoa!
NARRATOR: And Watson
pulls out other famous sci-fi flicks,
like Blade Runner.
It generates hundreds
of possible answers
NARRATOR: With
hundreds of choices,
how can Watson pick the
one answer that's correct?
FERRUCCI: Next thing
that Watson is going to do,
it's going to take
those answers and say,
"Well, let's assume all
of them might be right"
So these are its
competing hypotheses
NARRATOR: Watson
starts considering evidence
for and against each candidate,
using rules like "a movie
is sometimes called a flick"
GONDEK: And we'll
look at things like,
well, it's looking for a flick
Is this candidate
answer a flick?
Is The Matrix a flick?
Yes
Is Speed a flick?
Yes
Is Keanu Reeves a flick?
No... so we're starting
to learn something
NARRATOR: Within a
matter of milliseconds,
Watson analyzes
every possible answer
in hundreds of different ways
and scores each
piece of evidence
behind every answer in the list
That's a lot of scores
GONDEK: The problem is, you
have all these different scorers
and they don't agree
Some of the scorers
are going to say
"The Matrix" is the right answer
Some of the scorers
are going to say
"Keanu Reeves"
is the right answer
Some are going to think
Matrix 2 is the right answer
NARRATOR: And
a lot of scorers think
"Blade Runner"
is the right answer,
because it shows up
so often as a sci-fi flick
So you need someone at the end
to listen to all
these different votes
and decide what's going
to be the best answer
NARRATOR: This is where
Watson's machine learning kicks in
Having studied thousands
of other Jeopardy! questions
and their correct answers,
Watson has learned
what evidence is
important and what's not
What machine learning
will start to do is learn
how to weigh them
differently and say,
weighing questions like this,
calling on a phone, not calling
on a phone... not so important
This other stuff Do I
have a sci-fi movie?
Is the person named a
character in that movie?
Very, very important
for questions like this
NARRATOR: In this case, he
successfully weighs the evidence
and identifies sci-fi
flicks from 1999,
starring Keanu Reeves
So he picks the one answer
matching all those elements...
"The Matrix."
Watson's elaborate
system doesn't always work,
but without machine learning,
he wouldn't stand a chance
Machine learning isn't
just important for Watson
It's driving a
revolution in computing
It plays a major role
in computer models that predict
the weather days in advance
And all those recommendations
you get from Amazon or Netflix?
No human is writing up rules
about your likes and dislikes
Instead, computers are
comparing your preferences
to millions of other
customers' and finding patterns
and learning about you
Today machine learning is
conquering many problems
once thought too
complex for computers,
like speech recognition
TOM MITCHELL:
In the earlier days,
people decided that they
would try to program computers
to recognize speech...
"Which word am I saying now?"
Pick up the big block
at the right side
NARRATOR: In the '60s,
this voice-directed block world
was the height of technology
Computers could be programmed
to recognize the audio signals
of specific words and phrases
Pick up every small block
NARRATOR: But they had to be
reprogrammed for every new speaker
because everyone's
speech is slightly different
Even though it's very easy
for you and I to recognize
the word "ice cube"
Ice cube
It's very difficult for us
to write down the rules
that would allow a computer
to look at the microphone signal
and see that it's "ice cube"
NARRATOR: But now,
computers are trained
with millions of examples
of human speech
MITCHELL: Here's the microphone
signal and this is the word "ice cube"
Ice cube
MITCHELL: Here's another one
Ice cube
You end up
with much more successful
speech recognition systems
NARRATOR: Today, speech
recognition software, though not perfect,
is remarkably accurate
and getting better all the time
All the ones we have today
are based on machine learning,
simply because
that works the best
NARRATOR: And some
programs are taking it a step further
Where do you come from?
NARRATOR: Not only
transcribing your speech
[speaking Chinese]
NARRATOR: but translating it
into a foreign language as well
COMPUTER: Shanghai
It's very nice to meet you
[computer translating
into Chinese]
NARRATOR: Every language
has so much ambiguity...
Double meanings and metaphors
What are your specials today?
NARRATOR: Accurate computer
translation seemed impossible
with the old
rules-based approach
[speaking Chinese]
COMPUTER: Today's
special is roast beef fried rice
All of these interactions
are so complex
that you couldn't in your
lifetime write all these rules
It's just too enormous,
too daunting an effort
My wife asked me
to buy some crackers
[speaking Japanese]
COMPUTER: What kind of crackers?
Rice crackers
NARRATOR: Alex Waibel fed
a computer millions of examples
of English text, together
with their translations,
into about a dozen
different languages
Now he's got a program that
can run on your phone or iPad
Do you often go shopping here?
NARRATOR: And
translates on the go
[computer voicing
translation in Japanese]
[speaking Japanese]
NARRATOR: Machine
learning has been so successful,
mastering more and more tasks
once only done well by humans
The computer program
translates between languages
NARRATOR: Some researchers
believe it may be a crucial building block
for making real
artificial intelligence
[computer voicing
translation in Japanese]
There are two ways
of building intelligence
You either know how
to write down the recipe
or you let it grow itself
And it's pretty clear
that we don't know
how to write down the recipe
Machine learning is all
about giving it the capability
to grow itself
NARRATOR: Some
people find the idea
of a machine that
can learn threatening
But when Watson's
having a bad day,
it's hard to imagine
him threatening anyone
Did you see the Daily Double?
No So we were way ahead,
we were almost locking out
CRAIN: The other Daily Double!
We get the Daily Double,
there's like two or three clues left
So what do we do?
We bet big so we
can lock 'em out
WATSON: I'll wager $5,200
[chuckling]: Well!
NARRATOR: The team is nervous
They know Watson will never
make the cut on Jeopardy!
Unless he can stop
making dumb mistakes
Here's your clue
It was about letters
and it was, "a woman
wrote to this '40s artist"
CRAIN [reading]:
We answered with "Rembrandt"
WATSON: Who is Rembrandt?
Really?
NARRATOR: Although
Watson recognizes most dates,
he doesn't know that
the '40s refer to the 1940s
It's a '40s artist, we
answered with Rembrandt
There's a time a time problem
We all know that it
was Jackson Pollock
NARRATOR: Watson loses that game
We got the Daily Double
wrong and Final Jeopardy!
NARRATOR: And his
opponents show him no mercy
We both beat him!
Good for you!
Humans! Whoo!
Sometimes the reason
something is the right answer
is very obvious to a human,
like, for example, it may be
asking for a "she" or a "he"
CRAIN [reading]:
Watson?
WATSON: Who is Richard Nixon?
[laughter]
Oh, here you go Right?
CRAIN: Patricia?
Who is Pat Nixon?
CRAIN: That's correct
Richard Nixon was
never a first lady
I want to understand
what went on there
Our new gender stuff
is not in the system
It's not in the system
CHRIS WELTY: People take offense
at being called the wrong gender
Watson doesn't care
about stuff like that
It's making
statistical judgments
based on how different pieces
of evidence have gone together
in questions and
answers that we've given it
CRAIN [reading]:
WATSON: What is
Jimmy Dur-an-tay?
More specific?
WATSON: Sorry, all I know
is, "What is Jimmy Dur-an-tay?"
Saying it slower
doesn't make it right
FERRUCCI: Now you hear how
Todd Crain makes fun of the computer?
You know, I had my kids
I had them sign the
confidentiality agreement
and come in and see a
couple of these games
and their comment was,
"Why does the host make
fun of Watson, Daddy?"
This "What are you doing?"
website's name also refers
to a type of nervous laugh
CRAIN: Watson?
WATSON: What is evil laugh?
Oh, no!
In terms of comedy duos,
he is the best straight
man in the business
You're gonna kick
yourself Twitter
CRAIN: Because he
doesn't he doesn't get it
He doesn't get why his
inappropriate answer is funny,
and you can't ask for
better writing than that
Once in a while, okay,
we can all take a joke,
but over and over again, and
Watson's defenseless, right?
So he's making
fun of and criticizing
a defenseless computer
that represents people with
real feelings, real families
All right, maybe if I
don't have any feelings,
my kids have feelings
NARRATOR: And feelings
are intensified by the fact
there's just one more month
before the Jeopardy! producers
will return to make a decision
Watson, you have control
NARRATOR: The pressure is
on to boost Watson's strengths
and eliminate his weaknesses
His strengths are obvious
He dominates when it comes
to purely factual questions
BROWN: Watson usually does
very well at these factoid questions,
looking up facts... history,
geography, entertainment
CRAIN [reading]:
CRAIN: Watson?
WATSON: Who is Victor Flemming?
CRAIN: That is correct
Watson You choose again
WATSON: For 1600, please
CRAIN [reading]:
Watson?
WATSON: Who is John Ford?
Good for 1600
WATSON: Who is Mike Nichols?
CRAIN: Good for 12
Last clue on the
board for $800 Here it is
[reading]:
Watson?
WATSON: What is
The Pink Panther?
I just want to check...
Your buzzers are working?
[contestants laugh]
Just wanted to check in
and wake you from your nap
NARRATOR: Watson
has made it into the cloud,
but nowhere near
championship level
Ferrucci must
come up with a plan
to somehow step
up his performance
before the Jeopardy!
Producers return
The team has spent
almost four years,
enormous intellectual
and emotional energy,
and tens of millions of
dollars on developing Watson
But all this effort
isn't just for a machine
that can play Jeopardy!
IBM has much bigger goals
FERRUCCI: I'm already
looking sort of beyond Jeopardy!
I'm thinking, where
can we go from here?
NARRATOR: Ferrucci imagines
a day when Watson might perform
much like the
computer in Star Trek.
The captain just starts
talking to the computer,
and they say, "Computer"
CAPTAIN KIRK: Computer
COMPUTER: Ready
Could a storm of such
magnitude cause a power surge
in the transporter circuits
FERRUCCI: It's an
information seeking tool
that's capable of
understanding your question
KIRK: creating a momentary
interdimensional contact
with a parallel universe?
FERRUCCI: And dialoguing with you
to make sure that you get what you want
COMPUTER: Affirmative
Do you have one of those?
I don't have one
of those, right?
NARRATOR: You don't need to be
a starship commander
to find this helpful
In a world
overflowing with data,
intelligent expert systems
that can answer vital questions
have been the Holy Grail
Now, we could be
close to a Watson, MD
Think of it this way
There's a bunch
of information...
New diagnosis, new treatment
options, new discoveries
Can anybody keep that
all in their head all at once?
NARRATOR: A machine that could
access and organize all that information
could help doctors
analyze symptoms
and wade through
piles of medical journals
FERRUCCI: It changes the paradigm
in which we work with computers
That's the vision
WATSON: How am I doing?
I hope we will have
a good game today,
but first I have
to test my voice
NARRATOR: Before
that vision can be fulfilled,
before Watson can even
compete on Jeopardy!,
the team needs to get
more bugs out of his system
One of Watson's most
embarrassing weaknesses
is he cannot hear
Instead, Watson receives
each Jeopardy! clue
as an electronic text message
at the same moment his
competitors see it on the board
As a result, he doesn't know
the other contestants' answers
CRAIN [reading]:
Bill?
What's a mosquito?
CRAIN: No
Watson?
WATSON: What is mosquito?
CRAIN [laughing]: No
Harvey?
What's a horsefly?
CRAIN: Yes... thank you
for not saying mosquito
Good job, good
for $2,000, Harvey
NARRATOR: Ferrucci and the
team have been working furiously
to boost Watson's performance
As part of their plan, Watson
will now receive correct answers
electronically after
they're revealed
If the fix works, it will
be in the nick of time
The Jeopardy!
producers are back,
and they're about to
determine Watson's fate
This was a measure of our
progress, and we wanted to hear,
"Yeah, you're there,
you've made it"
NARRATOR: The new and
improved Watson gets his first big test
with a category called
"Celebrations of the Month"
CRAIN [reading]:
Watson?
WATSON: What is holiday?
No That's not even close, really
NARRATOR: Watson
fails, because he doesn't get
that in this category,
the answer must be
the name of a month...
Something his human
competition quickly figures out
CRAIN: Arthur?
ARTHUR: What is April?
CRAIN: What is April?
April 18th is
We don't understand the question
We don't understand
the category, basically
CRAIN [reading]:
NARRATOR: But now
he electronically receives
these correct responses
CRAIN: Arthur? What is June?
Good for 200
NARRATOR: Can he
learn from the answers?
GONDEK: What
Watson does here is,
it sees that all the answers
I've seen have been the month
in which this thing
in the clue occurs
CRAIN: Matt
What is November?
Good for four!
So then it knows in the
next clue to look for, um,
what month does
this thing occur in?
MATT: Celebrations for six
CRAIN: Celebrations for six
[reading]:
Watson?
WATSON: What is May?
GONDEK: Yes! He got it!
CRAIN: Very nicely done, Watson
He figured it out
It took us four
Yeah, yeah
NARRATOR: Thanks
to the team's efforts,
Watson is soaring
higher in the cloud
and now approaching the level
of champions like Ken Jennings
Watson is on a roll
Choose again
WATSON: "You're
Tripping," for 1600
CRAIN [reading]:
Watson?
WATSON: What is Montreal?
Yes
WATSON: Who is Zebulon Pike?
CRAIN: Good
WATSON: What is Providence
Aquarius Texas Beau Brummell?
CRAIN: Watson?
WATSON: What is "Ciao"?
That was right and
cute all at the same time
NARRATOR: The Jeopardy!
executives have seen enough
HARRY FRIEDMAN:
I think we've gone
from impressed to blown away
Very nicely done, Watson
NARRATOR: Finally, Watson
will get his chance on Jeopardy!
A computer playing
against human champions
ALEX TREBEK:
They say it can think
NARRATOR: In a game that
is a very symbol of intelligence
But can it think like a
Jeopardy! champion?
NARRATOR: It will be a
contest the world has never seen
Good luck, Watson
NARRATOR: But does this mean
that the dream of artificial
intelligence is coming true?
BROOKS: So,
artificial intelligence,
to me, is trying to get
computers to do stuff
that if people did them,
you'd say, "Oh, they're
demonstrating their peopleness
That's what makes humans
humans, that stuff they're doing"
NARRATOR: But without
experience or emotion,
can a computer like Watson ever
learn and understand the world
the way we humans do
from early childhood on?
Right now, no
machine can understand
the meaning of a play,
what it means to be King
Lear or Macbeth or Hamlet
LAURENCE OLIVIER:
To be or not to be
That is the question
No machine can understand
the parables of the
Judeo-Christian Bible
All they can do is
grovel through data
and find regularities
NARRATOR: But for Dave Ferrucci,
that kind of understanding
was never the goal
It's not going to
emerge as a human,
because it doesn't connect the
information to human experience,
to human cognition
When you think about
a great symphony,
and when a human
sits down with that,
that music is affecting that
human at an emotional level
The computer doesn't
have that human experience,
doesn't have that human emotion
It's not human, it's a computer
NARRATOR: Watson may never
experience the world the way we do,
but with his enormous
knowledge base,
his skill at
interpreting language
and his ability to learn
Yes, he got it!
WATSON: What is May?
He figured it out
NARRATOR: Could he actually
be considered "intelligent"?
Oh my God
It is more intelligent than
the average Jeopardy! player
in answering Jeopardy! Questions
That's impressively intelligent
WATSON: Good
afternoon, Mr. Trebek
I've been waiting
for this moment
for a very, very, very long time
NARRATOR:
Finally, it's show time
Watson is now taking the stage,
where his intelligence will
be put to the ultimate test...
In front of millions
of Jeopardy! viewers
GONDEK: I think I dream
about Jeopardy! questions now
I have nightmares about
Jeopardy! questions
NARRATOR: Has
the team done enough?
They're about to find out
as Watson meets the world's
two best Jeopardy! players,
Brad Rutter and Ken Jennings
GONDEK: We've never
had this caliber of a player
And there's a reason why
Ken won 74 games in a row;
there's a reason why
Brad has never been beat
by a human
NARRATOR: The whole team has
been waiting four years for this moment
Let's play Jeopardy!
Here we go
[reading]:
TREBEK: Watson
WATSON: What is shoe?
TREBEK: You are
right You get to pick
NARRATOR: At first, man and
machine look evenly matched
TREBEK: Watson
WATSON: Who is Jude?
TREBEK: Yes Brad
What is the 19-aughts
or the 1900s?
TREBEK: Yes Watson
What is The Last Judgment?
Correct Ken
What is Greece?
Greece... yes
NARRATOR: At the
end of the first round,
it's anyone's game
TREBEK: Sauron is right
That puts you into
a tie for the lead
The first round, 5,000, 5,000
They could beat us
We've seen it happen
in sparring games
Well, certainly if you wanted
a cliffhanger after the
first game, it's going well
Me? Not so much
NARRATOR: But as the match
continues, Watson dominates,
consistently beating
the humans to the buzz
TREBEK: Watson
WATSON: Who is Isaac Newton?
TREBEK: You are right
WATSON: Who is C S Lewis?
Yes
WATSON: What is guitar?
Right
NARRATOR: It starts
to look like a blowout
With that, you move to 36,681
NARRATOR: When, in the last
round, Ken Jennings stages a comeback
TREBEK [reading]:
Ken?
What is USA Today?
TREBEK: Right Ken?
What are national borders? Good
Martin Luther King
Yes, you're right [applause]
NARRATOR: Going into "Final
Jeopardy!" anything can happen
We just want to avoid a
stupid answer at this point
Here is the clue:
[reading]:
TREBEK: "Who is Bram Stoker?"
You are correct
The author of Dracula.
Over to Ken Jennings now
And we find, "Who is Stoker?"
"I for one welcome our
new computer overlord"
[laughter and applause]
NARRATOR: If Watson
gets this one, he will win
If we get "Final Jeopardy!"
right, I'm gonna kiss Eric
[laughter]
TREBEK: Now we come to Watson
We're looking for "Bram Stoker"
And we find, "Who
is Bram Stoker?"
[applause and cheering]
GONDEK: It's kind
of overwhelming
We won
And I think people will say,
"Yup, Watson was
the best player there"
NARRATOR: No question, Watson's
victory is a major milestone for AI,
but perhaps an even
greater achievement
for the human intelligence
that created him
MAN: Mission accomplished
[applause continues]
The exploration continues
on NOVA's website,
where you can find a Q&A
with Watson team
leader David Ferrucci,
see other smart machines
transforming our world,
and hear what top computer
scientists have to say
about the future of
artificial intelligence.
Dig deeper into
technology and engineering
with expert interviews,
interactives, teacher resources
and more.
Welcome to our World! Where we serve you cookies to ensure you get the best viewing experience on our site.
Did you know that you can remove censorship board-wide, use our advanced search functions, be notified when new content is posted, join our memberships, set episodes to show in any order you want & more if you are logged into your account?
Register or sign in here: ucp.php?mode=register
Did you know that you can remove censorship board-wide, use our advanced search functions, be notified when new content is posted, join our memberships, set episodes to show in any order you want & more if you are logged into your account?
Register or sign in here: ucp.php?mode=register
38x13 - Smartest Machine on Earth
Watch/Buy Amazon
Nova often includes interviews with scientists doing research in the subject areas covered and occasionally includes footage of a particular discovery.
Nova often includes interviews with scientists doing research in the subject areas covered and occasionally includes footage of a particular discovery.