Watson (artificial intelligence software)
Encyclopedia
Watson is an artificial intelligence
computer system capable of answering questions posed in natural language
, developed in IBM
's DeepQA project by a research team led by principal investigator
David Ferrucci. Watson was named after IBM's first president, Thomas J. Watson
.
In 2011, as a test of its abilities, Watson competed on the quiz show
Jeopardy!
, in the show's only human-versus-machine match-up to date. In a two-game, combined-point match, broadcast in three Jeopardy! episodes February 14–16, Watson beat Brad Rutter
, the biggest all-time money winner on Jeopardy!, and Ken Jennings
, the record holder for the longest championship streak (75 days). Watson received the first prize of $1 million, while Ken Jennings and Brad Rutter received $300,000 and $200,000, respectively. Jennings and Rutter pledged to donate half their winnings to charity, while IBM divided Watson's winnings between two charities.
Watson consistently outperformed its human opponents on the game's signaling device, but had trouble responding to a few categories, notably those having short clues containing only a few words. For each clue, Watson's three most probable responses were displayed by the television screen. Watson had access to 200 million pages of structured and unstructured content consuming four terabytes of disk storage, including the full text of Wikipedia
. Watson was not connected to the Internet during the game.
. IBM describes it as "an application of advanced Natural Language Processing
, Information Retrieval
, Knowledge Representation
and Reasoning
, and Machine Learning
technologies to the field of open domain question answering" which is "built on IBM's DeepQA technology for hypothesis generation, massive evidence gathering, analysis, and scoring."
According to John Rennie
, Watson can process 500 gigabytes, the equivalent of a million books, per second. IBM's master inventor
and senior consultant Tony Pearson estimated Watson's hardware cost at about $3 million and with 80 TeraFLOPs would be placed 94th on the Top 500 Supercomputers list, and 49th in the Top 50 Supercomputers list. According to Rennie, the content was stored in Watson's RAM for the game because data stored on hard drives are too slow to access.
framework for distributed computing, Apache UIMA
(Unstructured Information Management Architecture) framework, IBM’s DeepQA software and SUSE Linux Enterprise Server 11
operating system. “[...] more than 100 different techniques are used to analyze natural language, identify sources, find and generate hypotheses, find and score evidence, and merge and rank hypotheses.”
, WordNet
, and Yago
were used.
The IBM team provided Watson with millions of documents, including dictionaries, encyclopedias, and other reference material that it could use to build its knowledge. Although Watson was not connected to the Internet during the game, it contained 200 million pages of structured and unstructured content consuming four terabytes of disk storage, including the full text of Wikipedia
.
read each clue in its entirety, then a light was lit as a "ready" signal; the first to activate their buzzer
button won the chance to respond. Watson received the clues as electronic texts at the same moment they were made visible to the human players. It would then parse the clues into different keywords and sentence fragments in order to find statistically related phrases. Watson's main innovation was not in the creation of a new algorithm for this operation but rather its ability to quickly execute thousands of proven language analysis algorithms simultaneously to find the correct answer. The more algorithms that find the same answer independently the more likely Watson is to be correct. Once Watson has a small number of potential solutions it is able to check against its database to ascertain whether the solution makes sense. In a sequence of 20 mock games human participants were able to use the average six to seven seconds that Watson needed to hear the clue and decide whether to signal for responding. During that time Watson also has to evaluate the response and determine whether it is sufficiently confident in the result to signal. Part of the system used to win the Jeopardy! contest was the electronic circuitry that receives the "ready" signal and then examined whether Watson's confidence level was great enough to activate the buzzer. Given the speed of this circuitry compared to the speed of human reaction times, Watson's reaction time was faster than the human contestants except when the human anticipated (instead of reacted to) the ready signal. After signaling, Watson speaks with an electronic voice and gives the responses in Jeopardy! question format. Watson's voice was synthesized from recordings that actor Jeff Woodman made for an IBM text-to-speech program in 2004.
The Jeopardy! staff used different means to notify Watson and the human players when to buzz, which was critical in many rounds. The humans were notified by a light, which took them tenths of a second to perceive
. Watson was notified by an electronic signal and could activate the buzzer within about eight milliseconds. The humans tried to compensate for the perception delay by anticipating the light, but the variation in the anticipation time was generally too great to fall within Watson's response time. Watson did not operate to anticipate the notification signal.
in chess in 1997, IBM had been on the hunt for a new challenge. In 2004, IBM Research manager Charles Lickel, over dinner with coworkers, noticed that the restaurant they were in had fallen silent. He soon discovered the cause of this evening hiatus: Ken Jennings
, who was then in the middle of his successful 74-game run on Jeopardy!. Nearly the entire restaurant had piled toward the televisions, mid-meal, to watch the phenomenon. Intrigued by the quiz show as a possible challenge for IBM, Lickel passed the idea on, and in 2005, IBM Research executive Paul Horn
backed Lickel up, pushing for someone in his department to take up the challenge of playing Jeopardy! with an IBM system. Though he initially had trouble finding any research staff willing to take on what looked to be a much more complex challenge than the wordless game of chess, eventually David Ferrucci took him up on the offer. In competitions managed by the United States government, Watson's predecessor, a system named Piquant, was usually able to respond correctly to only about 35% of clues and often required several minutes to respond. To compete successfully on Jeopardy!, Watson would need to respond in no more than a few seconds, and at that time, the problems posed by the game show were deemed to be impossible to solve.
In initial tests run during 2006 by David Ferrucci, the senior manager of IBM's Semantic Analysis and Integration department, Watson was given 500 clues from past Jeopardy! programs. While the best real-life competitors buzzed in half the time and responded correctly to as many as 95% of clues, Watson's first pass could only get about 15% correct. During 2007, the IBM team was given three to five years and a staff of 15 people to solve the problems. By 2008, the developers had advanced Watson such that it could compete with Jeopardy! champions. By February 2010, Watson could beat human Jeopardy! contestants on a regular basis.
While primarily an IBM effort, Watson's development team includes faculty and students from Carnegie Mellon University
, University of Massachusetts Amherst
, University of Southern California
's Information Sciences Institute
, University of Texas at Austin
, Massachusetts Institute of Technology
, University of Trento
, and Rensselaer Polytechnic Institute
.
about the possibility of having Watson compete against Ken Jennings
and Brad Rutter
, two of the most successful contestants on the show, and the program's producers agreed. Watson's differences with human players had generated conflicts between IBM and Jeopardy! staff during the planning of the competition. IBM repeatedly expressed concerns that the show's writers would exploit Watson's cognitive deficiencies when writing the clues, thereby turning the game into a Turing test
. To alleviate that claim, a third party randomly picked the clues from previously written shows that were never broadcast. Jeopardy! staff also showed concerns over Watson's reaction time on the buzzer. Originally Watson signaled electronically, but show staff requested that it press a button physically, as the human contestants would. Even with a robotic "finger" pressing the buzzer, Watson remained faster than its human competitors. Ken Jennings noted, "If you're trying to win on the show, the buzzer is all," and that Watson "can knock out a microsecond-precise buzz every single time with little or no variation. Human reflexes can't compete with computer circuits in this regard." Stephen Baker
, a journalist who recorded Watson's development in his book Final Jeopardy, reported that the conflict between IBM and Jeopardy! became so serious in May of 2010 that the competition was almost canceled.
As part of the preparation, IBM constructed a mock set in a conference room at one of its technology sites to model the one used on Jeopardy! Human players, including former Jeopardy! contestants, also participated in mock games against Watson with Todd Alan Crain of The Onion
playing host. About 100 test matches were conducted with Watson winning 65% of the games.
To provide a physical presence in the televised games, Watson was represented by an "avatar
" of a globe, inspired by the IBM "smarter planet" symbol. Forty-two colored threads criss-crossed the globe, to represent Watson's state of thought; the number 42 was an in-joke referring to the novel The Hitchhiker's Guide to the Galaxy. Joshua Davis
, the artist who designed the avatar for the project, explained to Stephen Baker that there are 36 triggerable states that Watson was able to use throughout the game to show its confidence in responding to a clue correctly; he had hoped to be able to find forty-two, to add another level to the Hitchhiker's Guide reference, but he was unable to pinpoint enough game states.
A practice match was recorded on January 13, 2011, and the official matches were recorded on January 14, 2011. All participants maintained secrecy about the outcome until the match was broadcast in February.
Watson's performance was characterized by some quirks. In one instance, Watson repeated a reworded version of an incorrect response offered by Jennings (Jennings said "What are the '20s?" in reference to the 1920s. Then Watson said "What is 1920s?") Because Watson could not recognize other contestants' responses, it did not know that Jennings had already given the same response. In another instance, Watson was initially given credit for a response of "What is leg?" after Jennings incorrectly responded "What is: he only had one hand?" to a clue about George Eyser
(The correct response was, "What is: he's missing a leg?"). Because Watson, unlike a human, could not have been responding to Jennings's mistake, it was decided that this response was incorrect. The broadcast version of the episode was edited to omit Trebek's original acceptance of Watson's response. Watson also demonstrated complex wagering strategies on the Daily Doubles, with one bet at $6,435 and another at $1,246. Gerald Tesauro, one of the IBM researchers who worked on Watson, explained that Watson's wagers were based on its confidence level for the category and a complex regression model
called the Game State Evaluator.
Watson took a commanding lead in Double Jeopardy!, correctly responding to both Daily Doubles. Watson responded to the second Daily Double correctly with a 32% confidence score.
Although it wagered only $947 on the clue, Watson was the only contestant to miss the Final Jeopardy! response in the category U.S. CITIES ("Its largest airport was named for a World War II hero; its second largest, for a World War II battle"). Rutter and Jennings gave the correct response of Chicago, but Watson's response was "What is Toronto
?????" Ferrucci offered reasons why Watson would appear to have guessed a Canadian
city: categories only weakly suggest the type of response desired, the phrase "U.S. city" didn't appear in the question, there are cities named Toronto in the U.S., and Toronto, Ontario has an American League
baseball team. Dr. Chris Welty, who also worked on Watson, suggested that it may not have been able to correctly parse the second part of the clue, "its second largest, for a World War II battle" (which was not a standalone clause despite it following a semicolon
, and required context to understand that it was referring to a second-largest airport). Eric Nyberg, a professor at Carnegie Mellon University and a member of the development team, stated that the error occurred because Watson does not possess the comparative knowledge to discard that potential response as not viable. Although not displayed to the audience as with non-Final Jeopardy! questions, Watson's second choice was Chicago. Both Toronto and Chicago were well below Watson's confidence threshold, at 14% and 11% respectively. (This lack of confidence was the reason for the multiple question marks in Watson's response.)
The game ended with Jennings with $4,800, Rutter with $10,400, and Watson with $35,734.
jacket to the recording of the second match.
In the first round, Jennings was finally able to choose a Daily Double clue, while Watson responded to one Daily Double clue incorrectly for the first time in the Double Jeopardy! Round. After the first round, Watson placed second for the first time in the competition after Rutter and Jennings were briefly successful in increasing their dollar values before Watson could respond. Nonetheless, the final result ended with a victory for Watson with a score of $77,147, besting Jennings who scored $24,000 and Rutter who scored $21,600.
and 50% going to World Community Grid
. Likewise, Jennings and Rutter donated 50% of their winnings to their respective charities.
In acknowledgment of IBM and Watson's achievements, Jennings made an additional remark in his Final Jeopardy! response: "I for one welcome our new computer overlords", echoing a similar memetic reference
to the episode "Deep Space Homer
" on The Simpsons
, in which TV news presenter Kent Brockman speaks of welcoming "our new insect overlords". Jennings later wrote an article for Slate
, in which he stated "IBM has bragged to the media that Watson's question-answering skills are good for more than annoying Alex Trebek. The company sees a future in which fields like medical diagnosis
, business analytics
, and tech support
are automated by question-answering software like Watson. Just as factory jobs were eliminated in the 20th century by new assembly-line robots, Brad and I were the first knowledge-industry workers
put out of work by the new generation of 'thinking' machines. 'Quiz show contestant' may be the first job made redundant by Watson, but I'm sure it won't be the last."
: Rush D. Holt, Jr.
(D-NJ, a former Jeopardy! contestant), Jim Himes
(D-CT), Jared Polis
(D-CO), Nan Hayworth
(R-NY) and Bill Cassidy
(R-LA). IBM organized the event to "foster a conversation about how technology can positively impact society".
In the only round he played, Holt led with Watson in second place. However, combining the scores between all matches, the final score was $40,300 for Watson and $30,000 for the congressional players combined.
IBM and Nuance Communications
Inc. are partnering for the research project to develop a commercial product during the next 18 to 24 months that will exploit Watson’s capabilities as a clinical decision support system
to aid the diagnosis and treatment of patients. Physicians at Columbia University
are helping identify critical issues in the practice of medicine where the Watson technology may be able to contribute and physicians at the University of Maryland
are working to identify the best way that a technology like Watson could interact with medical practitioners to provide the maximum assistance. It has been also suggested by Robert C. Weber, IBM's general counsel
, that Watson may be used for legal research.
Watson is based on commercially available IBM Power 750 servers that have been marketed since February 2010. IBM also intends to market the DeepQA software to large corporations, with a price in the millions of dollars, reflecting the $1 million needed to acquire a server that meets the minimum system requirement to operate Watson. IBM expects the price to decrease substantially within a decade as the technology improves.
Commentator Rick Merritt said that "there's another really important reason why it is strategic for IBM to be seen very broadly by the American public as a company that can tackle tough computer problems. A big slice of Big Blue's pie comes from selling to the U.S. government some of the biggest, most expensive systems in the world."
argues that Watson—despite impressive capabilities—could not actually think. Drawing on his Chinese room
thought experiment, Searle claims that Watson, like other computational machines, is only capable of manipulating symbols, but has no ability to understand the meaning of those symbols; however, Searle's experiment may be flawed.
Artificial intelligence
Artificial intelligence is the intelligence of machines and the branch of computer science that aims to create it. AI textbooks define the field as "the study and design of intelligent agents" where an intelligent agent is a system that perceives its environment and takes actions that maximize its...
computer system capable of answering questions posed in natural language
Natural language
In the philosophy of language, a natural language is any language which arises in an unpremeditated fashion as the result of the innate facility for language possessed by the human intellect. A natural language is typically used for communication, and may be spoken, signed, or written...
, developed in IBM
IBM
International Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas...
's DeepQA project by a research team led by principal investigator
Principal investigator
A principal investigator is the lead scientist or engineer for a particular well-defined science project, such as a laboratory study or clinical trial....
David Ferrucci. Watson was named after IBM's first president, Thomas J. Watson
Thomas J. Watson
Thomas John Watson, Sr. was president of International Business Machines , who oversaw that company's growth into an international force from 1914 to 1956...
.
In 2011, as a test of its abilities, Watson competed on the quiz show
Quiz Show
Quiz Show is a 1994 American historical drama film produced and directed by Robert Redford. Adapted by Paul Attanasio from Richard Goodwin's memoir Remembering America, the film is based upon the Twenty One quiz show scandal of the 1950s...
Jeopardy!
Jeopardy!
Griffin's first conception of the game used a board comprising ten categories with ten clues each, but after finding that this board could not be shown on camera easily, he reduced it to two rounds of thirty clues each, with five clues in each of six categories...
, in the show's only human-versus-machine match-up to date. In a two-game, combined-point match, broadcast in three Jeopardy! episodes February 14–16, Watson beat Brad Rutter
Brad Rutter
Bradford Gates "Brad" Rutter is the biggest all-time money winner on the U.S. syndicated game show Jeopardy! and the second biggest all-time money winner on a game show....
, the biggest all-time money winner on Jeopardy!, and Ken Jennings
Ken Jennings
Kenneth Wayne "Ken" Jennings III is an American game show contestant and author. Jennings is noted for holding the record for the longest winning streak on the U.S. syndicated game show Jeopardy! and as being the all-time leading money winner on American game shows...
, the record holder for the longest championship streak (75 days). Watson received the first prize of $1 million, while Ken Jennings and Brad Rutter received $300,000 and $200,000, respectively. Jennings and Rutter pledged to donate half their winnings to charity, while IBM divided Watson's winnings between two charities.
Watson consistently outperformed its human opponents on the game's signaling device, but had trouble responding to a few categories, notably those having short clues containing only a few words. For each clue, Watson's three most probable responses were displayed by the television screen. Watson had access to 200 million pages of structured and unstructured content consuming four terabytes of disk storage, including the full text of Wikipedia
Wikipedia
Wikipedia is a free, web-based, collaborative, multilingual encyclopedia project supported by the non-profit Wikimedia Foundation. Its 20 million articles have been written collaboratively by volunteers around the world. Almost all of its articles can be edited by anyone with access to the site,...
. Watson was not connected to the Internet during the game.
Architecture
Watson is a question answering (QA) computing system built by IBMIBM
International Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas...
. IBM describes it as "an application of advanced Natural Language Processing
Natural language processing
Natural language processing is a field of computer science and linguistics concerned with the interactions between computers and human languages; it began as a branch of artificial intelligence....
, Information Retrieval
Information retrieval
Information retrieval is the area of study concerned with searching for documents, for information within documents, and for metadata about documents, as well as that of searching structured storage, relational databases, and the World Wide Web...
, Knowledge Representation
Knowledge representation
Knowledge representation is an area of artificial intelligence research aimed at representing knowledge in symbols to facilitate inferencing from those knowledge elements, creating new elements of knowledge...
and Reasoning
Automated reasoning
Automated reasoning is an area of computer science dedicated to understand different aspects of reasoning. The study in automated reasoning helps produce software which allows computers to reason completely, or nearly completely, automatically...
, and Machine Learning
Machine learning
Machine learning, a branch of artificial intelligence, is a scientific discipline concerned with the design and development of algorithms that allow computers to evolve behaviors based on empirical data, such as from sensor data or databases...
technologies to the field of open domain question answering" which is "built on IBM's DeepQA technology for hypothesis generation, massive evidence gathering, analysis, and scoring."
Hardware
According to IBM:Watson is a workload optimized system designed for complex analyticsAnalyticsAnalytics is the application of computer technology, operational research, and statistics to solve problems in business and industry. Analytics is carried out within an information system: while, in the past, statistics and mathematics could be studied without computers and software, analytics has...
, made possible by integrating massively parallel POWER7POWER7POWER7 is a Power Architecture microprocessor released in 2010 that succeeded the POWER6. POWER7 was developed by IBM at several sites including IBM's Rochester, MN; Austin, TX; Essex Junction, Vermont; T. J. Watson Research Center, NY; Bromont, QC and Böblingen, Germany laboratories...
processors and the IBM DeepQA software to answer Jeopardy!Jeopardy!Griffin's first conception of the game used a board comprising ten categories with ten clues each, but after finding that this board could not be shown on camera easily, he reduced it to two rounds of thirty clues each, with five clues in each of six categories...
questions in under three seconds. Watson is made up of a cluster of ninety IBM Power 750 servers (plus additional I/O, network and cluster controller nodes in 10 racks) with a total of 2880 POWER7 processor cores and 16 TerabyteTerabyteThe terabyte is a multiple of the unit byte for digital information. The prefix tera means 1012 in the International System of Units , and therefore 1 terabyte is , or 1 trillion bytes, or 1000 gigabytes. 1 terabyte in binary prefixes is 0.9095 tebibytes, or 931.32 gibibytes...
s of RAM. Each Power 750 server uses a 3.5 GHz POWER7 eight core processor, with four threads per core. The POWER7 processor's massively parallel processing capability is an ideal match for Watson's IBM DeepQA software which is embarrassingly parallelEmbarrassingly parallelIn parallel computing, an embarrassingly parallel workload is one for which little or no effort is required to separate the problem into a number of parallel tasks...
(that is a workload that is easily split up into multiple parallel tasks).
According to John Rennie
John Rennie (editor)
John Rennie was the seventh editor in chief of Scientific American magazine. He holds a bachelors of science degree in biology from Yale University...
, Watson can process 500 gigabytes, the equivalent of a million books, per second. IBM's master inventor
IBM Master Inventor
An IBM Master Inventor is an individual selected by IBM. The qualification and appointment to Master Inventor is governed by the value to IBM of an inventor's contribution to the patent portfolio, and through defensive publication...
and senior consultant Tony Pearson estimated Watson's hardware cost at about $3 million and with 80 TeraFLOPs would be placed 94th on the Top 500 Supercomputers list, and 49th in the Top 50 Supercomputers list. According to Rennie, the content was stored in Watson's RAM for the game because data stored on hard drives are too slow to access.
Software
Watson's software was written in various languages, including at least Java, C++, and Prolog and uses Apache HadoopHadoop
Apache Hadoop is a software framework that supports data-intensive distributed applications under a free license. It enables applications to work with thousands of nodes and petabytes of data...
framework for distributed computing, Apache UIMA
Uima
UIMA stands for Unstructured Information Management Architecture. An OASIS standard as of March 2009, UIMA is to date the only industry standard for content analytics....
(Unstructured Information Management Architecture) framework, IBM’s DeepQA software and SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server
SUSE Linux Enterprise Server is a Linux distribution supplied by SUSE and targeted at the business market. It is targeted for servers, mainframes, and workstations but can be installed on desktop computers for testing as well. New major versions are released at an interval of 3-4 years, while...
operating system. “[...] more than 100 different techniques are used to analyze natural language, identify sources, find and generate hypotheses, find and score evidence, and merge and rank hypotheses.”
Data
The sources of information for Watson include encyclopedias, dictionaries, thesauri, newswire articles, and literary works. Watson also used databases, taxonomies, and ontologies. Specifically, DBPediaDBpedia
DBpedia is a project aiming to extract structured content from the information created as part of the Wikipedia project. This structured information is then made available on the World Wide Web. DBpedia allows users to query relationships and properties associated with Wikipedia resources,...
, WordNet
WordNet
WordNet is a lexical database for the English language. It groups English words into sets of synonyms called synsets, provides short, general definitions, and records the various semantic relations between these synonym sets...
, and Yago
YAGO (database)
YAGO is a knowledge base developed at the Max-Planck-Institute Saarbrücken.The knowledge base contains information harvested from Wikipedia and linked to Wordnet....
were used.
The IBM team provided Watson with millions of documents, including dictionaries, encyclopedias, and other reference material that it could use to build its knowledge. Although Watson was not connected to the Internet during the game, it contained 200 million pages of structured and unstructured content consuming four terabytes of disk storage, including the full text of Wikipedia
Wikipedia
Wikipedia is a free, web-based, collaborative, multilingual encyclopedia project supported by the non-profit Wikimedia Foundation. Its 20 million articles have been written collaboratively by volunteers around the world. Almost all of its articles can be edited by anyone with access to the site,...
.
Operation
When playing Jeopardy! all players, including Watson, had to wait until host Alex TrebekAlex Trebek
George Alexander "Alex" Trebek is a Canadian American game show host who has been the host of the game show Jeopardy! since 1984, and prior to that, he hosted game shows such as Pitfall and High Rollers. He has appeared in numerous television series, usually as himself...
read each clue in its entirety, then a light was lit as a "ready" signal; the first to activate their buzzer
Buzzer
A buzzer or beeper is an audio signaling device, which may be mechanical, electromechanical, or piezoelectric. Typical uses of buzzers and beepers include alarm devices, timers and confirmation of user input such as a mouse click or keystroke....
button won the chance to respond. Watson received the clues as electronic texts at the same moment they were made visible to the human players. It would then parse the clues into different keywords and sentence fragments in order to find statistically related phrases. Watson's main innovation was not in the creation of a new algorithm for this operation but rather its ability to quickly execute thousands of proven language analysis algorithms simultaneously to find the correct answer. The more algorithms that find the same answer independently the more likely Watson is to be correct. Once Watson has a small number of potential solutions it is able to check against its database to ascertain whether the solution makes sense. In a sequence of 20 mock games human participants were able to use the average six to seven seconds that Watson needed to hear the clue and decide whether to signal for responding. During that time Watson also has to evaluate the response and determine whether it is sufficiently confident in the result to signal. Part of the system used to win the Jeopardy! contest was the electronic circuitry that receives the "ready" signal and then examined whether Watson's confidence level was great enough to activate the buzzer. Given the speed of this circuitry compared to the speed of human reaction times, Watson's reaction time was faster than the human contestants except when the human anticipated (instead of reacted to) the ready signal. After signaling, Watson speaks with an electronic voice and gives the responses in Jeopardy! question format. Watson's voice was synthesized from recordings that actor Jeff Woodman made for an IBM text-to-speech program in 2004.
Comparison with human players
Watson's basic working principle is to parse keywords in a clue while searching for related terms as responses. This gives Watson some advantages and disadvantages compared with human Jeopardy! players. Watson has deficiencies in understanding the contexts of the clues. As a result, human players usually generate responses faster than Watson, especially to short clues. Watson's programming prevents it from using the popular tactic of buzzing before it is sure of its response. Watson has consistently better reaction time on the buzzer once it has generated a response, and is immune to human players' psychological tactics.The Jeopardy! staff used different means to notify Watson and the human players when to buzz, which was critical in many rounds. The humans were notified by a light, which took them tenths of a second to perceive
Perception
Perception is the process of attaining awareness or understanding of the environment by organizing and interpreting sensory information. All perception involves signals in the nervous system, which in turn result from physical stimulation of the sense organs...
. Watson was notified by an electronic signal and could activate the buzzer within about eight milliseconds. The humans tried to compensate for the perception delay by anticipating the light, but the variation in the anticipation time was generally too great to fall within Watson's response time. Watson did not operate to anticipate the notification signal.
Development history
Since Deep Blue's victory over Garry KasparovGarry Kasparov
Garry Kimovich Kasparov is a Russian chess grandmaster, a former World Chess Champion, writer, political activist, and one of the greatest chess players of all time....
in chess in 1997, IBM had been on the hunt for a new challenge. In 2004, IBM Research manager Charles Lickel, over dinner with coworkers, noticed that the restaurant they were in had fallen silent. He soon discovered the cause of this evening hiatus: Ken Jennings
Ken Jennings
Kenneth Wayne "Ken" Jennings III is an American game show contestant and author. Jennings is noted for holding the record for the longest winning streak on the U.S. syndicated game show Jeopardy! and as being the all-time leading money winner on American game shows...
, who was then in the middle of his successful 74-game run on Jeopardy!. Nearly the entire restaurant had piled toward the televisions, mid-meal, to watch the phenomenon. Intrigued by the quiz show as a possible challenge for IBM, Lickel passed the idea on, and in 2005, IBM Research executive Paul Horn
Paul Horn (computer scientist)
Dr. Paul M. Horn is an American computer scientist and solid state physicist who has made contributions to pervasive computing, pioneered the use of copper and self-assembly in chip manufacturing, and he helped manage the development of deep computing, an important tool that provides business...
backed Lickel up, pushing for someone in his department to take up the challenge of playing Jeopardy! with an IBM system. Though he initially had trouble finding any research staff willing to take on what looked to be a much more complex challenge than the wordless game of chess, eventually David Ferrucci took him up on the offer. In competitions managed by the United States government, Watson's predecessor, a system named Piquant, was usually able to respond correctly to only about 35% of clues and often required several minutes to respond. To compete successfully on Jeopardy!, Watson would need to respond in no more than a few seconds, and at that time, the problems posed by the game show were deemed to be impossible to solve.
In initial tests run during 2006 by David Ferrucci, the senior manager of IBM's Semantic Analysis and Integration department, Watson was given 500 clues from past Jeopardy! programs. While the best real-life competitors buzzed in half the time and responded correctly to as many as 95% of clues, Watson's first pass could only get about 15% correct. During 2007, the IBM team was given three to five years and a staff of 15 people to solve the problems. By 2008, the developers had advanced Watson such that it could compete with Jeopardy! champions. By February 2010, Watson could beat human Jeopardy! contestants on a regular basis.
While primarily an IBM effort, Watson's development team includes faculty and students from Carnegie Mellon University
Carnegie Mellon University
Carnegie Mellon University is a private research university in Pittsburgh, Pennsylvania, United States....
, University of Massachusetts Amherst
University of Massachusetts Amherst
The University of Massachusetts Amherst is a public research and land-grant university in Amherst, Massachusetts, United States and the flagship of the University of Massachusetts system...
, University of Southern California
University of Southern California
The University of Southern California is a private, not-for-profit, nonsectarian, research university located in Los Angeles, California, United States. USC was founded in 1880, making it California's oldest private research university...
's Information Sciences Institute
Information Sciences Institute
The Information Sciences Institute is a research and development unit of the University of Southern California's Viterbi School of Engineering which focuses on computer and communications technology and information processing...
, University of Texas at Austin
University of Texas at Austin
The University of Texas at Austin is a state research university located in Austin, Texas, USA, and is the flagship institution of the The University of Texas System. Founded in 1883, its campus is located approximately from the Texas State Capitol in Austin...
, Massachusetts Institute of Technology
Massachusetts Institute of Technology
The Massachusetts Institute of Technology is a private research university located in Cambridge, Massachusetts. MIT has five schools and one college, containing a total of 32 academic departments, with a strong emphasis on scientific and technological education and research.Founded in 1861 in...
, University of Trento
University of Trento
The University of Trento is an Italian university located in the cities of Trento and Rovereto. It has been able to achieve considerable results in didactics, research and international relations, as shown by Censis University Guide and by the Italian Ministry of...
, and Rensselaer Polytechnic Institute
Rensselaer Polytechnic Institute
Stephen Van Rensselaer established the Rensselaer School on November 5, 1824 with a letter to the Rev. Dr. Samuel Blatchford, in which van Rensselaer asked Blatchford to serve as the first president. Within the letter he set down several orders of business. He appointed Amos Eaton as the school's...
.
Preparation
In 2008, IBM representatives communicated with Jeopardy! executive producer Harry FriedmanHarry Friedman
Harry Friedman is an American television industry executive. He has been the executive producer of America's highest-rated syndicated game shows, Jeopardy! and Wheel of Fortune, since 1999....
about the possibility of having Watson compete against Ken Jennings
Ken Jennings
Kenneth Wayne "Ken" Jennings III is an American game show contestant and author. Jennings is noted for holding the record for the longest winning streak on the U.S. syndicated game show Jeopardy! and as being the all-time leading money winner on American game shows...
and Brad Rutter
Brad Rutter
Bradford Gates "Brad" Rutter is the biggest all-time money winner on the U.S. syndicated game show Jeopardy! and the second biggest all-time money winner on a game show....
, two of the most successful contestants on the show, and the program's producers agreed. Watson's differences with human players had generated conflicts between IBM and Jeopardy! staff during the planning of the competition. IBM repeatedly expressed concerns that the show's writers would exploit Watson's cognitive deficiencies when writing the clues, thereby turning the game into a Turing test
Turing test
The Turing test is a test of a machine's ability to exhibit intelligent behaviour. In Turing's original illustrative example, a human judge engages in a natural language conversation with a human and a machine designed to generate performance indistinguishable from that of a human being. All...
. To alleviate that claim, a third party randomly picked the clues from previously written shows that were never broadcast. Jeopardy! staff also showed concerns over Watson's reaction time on the buzzer. Originally Watson signaled electronically, but show staff requested that it press a button physically, as the human contestants would. Even with a robotic "finger" pressing the buzzer, Watson remained faster than its human competitors. Ken Jennings noted, "If you're trying to win on the show, the buzzer is all," and that Watson "can knock out a microsecond-precise buzz every single time with little or no variation. Human reflexes can't compete with computer circuits in this regard." Stephen Baker
Stephen Baker
Stephen Edward Baker is a former professional American football player and a current New Jersey High School Track and Field Coach.-Professional career:...
, a journalist who recorded Watson's development in his book Final Jeopardy, reported that the conflict between IBM and Jeopardy! became so serious in May of 2010 that the competition was almost canceled.
As part of the preparation, IBM constructed a mock set in a conference room at one of its technology sites to model the one used on Jeopardy! Human players, including former Jeopardy! contestants, also participated in mock games against Watson with Todd Alan Crain of The Onion
The Onion
The Onion is an American news satire organization. It is an entertainment newspaper and a website featuring satirical articles reporting on international, national, and local news, in addition to a non-satirical entertainment section known as The A.V. Club...
playing host. About 100 test matches were conducted with Watson winning 65% of the games.
To provide a physical presence in the televised games, Watson was represented by an "avatar
Avatar (computing)
In computing, an avatar is the graphical representation of the user or the user's alter ego or character. It may take either a three-dimensional form, as in games or virtual worlds, or a two-dimensional form as an icon in Internet forums and other online communities. It can also refer to a text...
" of a globe, inspired by the IBM "smarter planet" symbol. Forty-two colored threads criss-crossed the globe, to represent Watson's state of thought; the number 42 was an in-joke referring to the novel The Hitchhiker's Guide to the Galaxy. Joshua Davis
Joshua Davis (web designer)
Joshua Davis is an American web designer, author and artist in new media. He was an early pioneer in the use of Macromedia Flash as a tool to generate Art...
, the artist who designed the avatar for the project, explained to Stephen Baker that there are 36 triggerable states that Watson was able to use throughout the game to show its confidence in responding to a clue correctly; he had hoped to be able to find forty-two, to add another level to the Hitchhiker's Guide reference, but he was unable to pinpoint enough game states.
A practice match was recorded on January 13, 2011, and the official matches were recorded on January 14, 2011. All participants maintained secrecy about the outcome until the match was broadcast in February.
Practice match
In a practice match before the press on January 13, 2011, Watson won a 15-question round against Ken Jennings and Brad Rutter with a score of $4,400 to Jennings's $3,400 and Rutter's $1,200, though Jennings and Watson were tied before the final $1,000 question. None of the three players responded incorrectly to a clue.First match
The first round was broadcast February 14, 2011. The right to choose the first category had been determined by a draw won by Rutter. Watson, represented by a computer monitor display and artificial voice, responded correctly to the second clue and then selected the fourth clue of the first category, a deliberate strategy to find the Daily Double as quickly as possible. Watson's guess at the Daily Double location was correct. At the end of the first round, Watson was tied with Rutter at $5,000; Jennings had $2,000.Watson's performance was characterized by some quirks. In one instance, Watson repeated a reworded version of an incorrect response offered by Jennings (Jennings said "What are the '20s?" in reference to the 1920s. Then Watson said "What is 1920s?") Because Watson could not recognize other contestants' responses, it did not know that Jennings had already given the same response. In another instance, Watson was initially given credit for a response of "What is leg?" after Jennings incorrectly responded "What is: he only had one hand?" to a clue about George Eyser
George Eyser
George Eyser was a German-American gymnast who competed in the 1904 Summer Olympics, earning six medals in one day, including three gold and two silver medals. Eyser competed with a wooden prosthesis for a left leg, having lost his real leg after being run over by a train...
(The correct response was, "What is: he's missing a leg?"). Because Watson, unlike a human, could not have been responding to Jennings's mistake, it was decided that this response was incorrect. The broadcast version of the episode was edited to omit Trebek's original acceptance of Watson's response. Watson also demonstrated complex wagering strategies on the Daily Doubles, with one bet at $6,435 and another at $1,246. Gerald Tesauro, one of the IBM researchers who worked on Watson, explained that Watson's wagers were based on its confidence level for the category and a complex regression model
Regression analysis
In statistics, regression analysis includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables...
called the Game State Evaluator.
Watson took a commanding lead in Double Jeopardy!, correctly responding to both Daily Doubles. Watson responded to the second Daily Double correctly with a 32% confidence score.
Although it wagered only $947 on the clue, Watson was the only contestant to miss the Final Jeopardy! response in the category U.S. CITIES ("Its largest airport was named for a World War II hero; its second largest, for a World War II battle"). Rutter and Jennings gave the correct response of Chicago, but Watson's response was "What is Toronto
Toronto
Toronto is the provincial capital of Ontario and the largest city in Canada. It is located in Southern Ontario on the northwestern shore of Lake Ontario. A relatively modern city, Toronto's history dates back to the late-18th century, when its land was first purchased by the British monarchy from...
?????" Ferrucci offered reasons why Watson would appear to have guessed a Canadian
Canada
Canada is a North American country consisting of ten provinces and three territories. Located in the northern part of the continent, it extends from the Atlantic Ocean in the east to the Pacific Ocean in the west, and northward into the Arctic Ocean...
city: categories only weakly suggest the type of response desired, the phrase "U.S. city" didn't appear in the question, there are cities named Toronto in the U.S., and Toronto, Ontario has an American League
American League
The American League of Professional Baseball Clubs, or simply the American League , is one of two leagues that make up Major League Baseball in the United States and Canada. It developed from the Western League, a minor league based in the Great Lakes states, which eventually aspired to major...
baseball team. Dr. Chris Welty, who also worked on Watson, suggested that it may not have been able to correctly parse the second part of the clue, "its second largest, for a World War II battle" (which was not a standalone clause despite it following a semicolon
Semicolon
The semicolon is a punctuation mark with several uses. The Italian printer Aldus Manutius the Elder established the practice of using the semicolon to separate words of opposed meaning and to indicate interdependent statements. "The first printed semicolon was the work of ... Aldus Manutius"...
, and required context to understand that it was referring to a second-largest airport). Eric Nyberg, a professor at Carnegie Mellon University and a member of the development team, stated that the error occurred because Watson does not possess the comparative knowledge to discard that potential response as not viable. Although not displayed to the audience as with non-Final Jeopardy! questions, Watson's second choice was Chicago. Both Toronto and Chicago were well below Watson's confidence threshold, at 14% and 11% respectively. (This lack of confidence was the reason for the multiple question marks in Watson's response.)
The game ended with Jennings with $4,800, Rutter with $10,400, and Watson with $35,734.
Second match
During the introduction, Trebek (a Canadian native) joked that he had learned Toronto was a U.S. city, and Watson's error in the first match prompted an IBM engineer to wear a Toronto Blue JaysToronto Blue Jays
The Toronto Blue Jays are a professional baseball team located in Toronto, Ontario, Canada. The Blue Jays are a member of the Eastern Division of Major League Baseball 's American League ....
jacket to the recording of the second match.
In the first round, Jennings was finally able to choose a Daily Double clue, while Watson responded to one Daily Double clue incorrectly for the first time in the Double Jeopardy! Round. After the first round, Watson placed second for the first time in the competition after Rutter and Jennings were briefly successful in increasing their dollar values before Watson could respond. Nonetheless, the final result ended with a victory for Watson with a score of $77,147, besting Jennings who scored $24,000 and Rutter who scored $21,600.
Final outcome
The prizes for the competition were $1 million for first place (Watson), $300,000 for second place (Jennings), and $200,000 for third place (Rutter). As promised, IBM donated 100% of Watson's winnings to charity, with 50% of those winnings going to World VisionWorld Vision
World Vision, founded in the USA in 1950, is an evangelical relief and development organization whose stated goal is "to follow our Lord and Saviour Jesus Christ in working with the poor and oppressed to promote human transformation, seek justice and bear witness to the good news of the Kingdom of...
and 50% going to World Community Grid
World Community Grid
World Community Grid is an effort to create the world's largest public computing grid to tackle scientific research projects that benefit humanity...
. Likewise, Jennings and Rutter donated 50% of their winnings to their respective charities.
In acknowledgment of IBM and Watson's achievements, Jennings made an additional remark in his Final Jeopardy! response: "I for one welcome our new computer overlords", echoing a similar memetic reference
Internet meme
The term Internet meme is used to describe a concept that spreads via the Internet. The term is a reference to the concept of memes, although the latter concept refers to a much broader category of cultural information.-Description:...
to the episode "Deep Space Homer
Deep Space Homer
"Deep Space Homer" is the fifteenth episode of The Simpsons fifth season and first aired on February 24, 1994. The episode was directed by Carlos Baeza and was the only episode of The Simpsons written by David Mirkin, who was also the executive producer at the time...
" on The Simpsons
The Simpsons
The Simpsons is an American animated sitcom created by Matt Groening for the Fox Broadcasting Company. The series is a satirical parody of a middle class American lifestyle epitomized by its family of the same name, which consists of Homer, Marge, Bart, Lisa and Maggie...
, in which TV news presenter Kent Brockman speaks of welcoming "our new insect overlords". Jennings later wrote an article for Slate
Slate (magazine)
Slate is a US-based English language online current affairs and culture magazine created in 1996 by former New Republic editor Michael Kinsley, initially under the ownership of Microsoft as part of MSN. On 21 December 2004 it was purchased by the Washington Post Company...
, in which he stated "IBM has bragged to the media that Watson's question-answering skills are good for more than annoying Alex Trebek. The company sees a future in which fields like medical diagnosis
Medical diagnosis
Medical diagnosis refers both to the process of attempting to determine or identify a possible disease or disorder , and to the opinion reached by this process...
, business analytics
Business analytics
Business analytics refers to the skills, technologies, applications and practices for continuous iterative exploration and investigation of past business performance to gain insight and drive business planning. Business analytics focuses on developing new insights and understanding of business...
, and tech support
Technical support
Technical support or tech support refers to a range of services by which enterprises provide assistance to users of technology products such as mobile phones, televisions, computers, software products or other electronic or mechanical goods...
are automated by question-answering software like Watson. Just as factory jobs were eliminated in the 20th century by new assembly-line robots, Brad and I were the first knowledge-industry workers
Knowledge worker
Knowledge workers in today's workforce are individuals who are valued for their ability to act and communicate with knowledge within a specific subject area. They will often advance the overall understanding of that subject through focused analysis, design and/or development. They use research...
put out of work by the new generation of 'thinking' machines. 'Quiz show contestant' may be the first job made redundant by Watson, but I'm sure it won't be the last."
Match against members of the United States Congress
On February 28, 2011, Watson played an untelevised exhibition match of Jeopardy! against five members of the United States House of RepresentativesUnited States House of Representatives
The United States House of Representatives is one of the two Houses of the United States Congress, the bicameral legislature which also includes the Senate.The composition and powers of the House are established in Article One of the Constitution...
: Rush D. Holt, Jr.
Rush D. Holt, Jr.
Rush Dew Holt, Jr. is the U.S. Representative for . He is a member of the Democratic Party. He is currently the only Quaker in Congress.-Early life and education :Rush D. Holt was born to Rush D...
(D-NJ, a former Jeopardy! contestant), Jim Himes
Jim Himes
James A. "Jim" Himes is an American businessman and U.S. Representative for , serving since 2009. He is a member of the Democratic Party....
(D-CT), Jared Polis
Jared Polis
Jared Schutz Polis is an American entrepreneur, philanthropist, and U.S. Representative for , serving since 2009. He is a member of the Democratic Party...
(D-CO), Nan Hayworth
Nan Hayworth
Nan Alison Sutter Hayworth is the U.S. Representative for . She is a member of the Republican Party.-Early life, education, and nursing career:...
(R-NY) and Bill Cassidy
Bill Cassidy
William "Bill" Cassidy is the U.S. Representative for , serving since 2009. He is a member of the Republican Party.-Early life, education and career:...
(R-LA). IBM organized the event to "foster a conversation about how technology can positively impact society".
In the only round he played, Holt led with Watson in second place. However, combining the scores between all matches, the final score was $40,300 for Watson and $30,000 for the congressional players combined.
Future uses
According to IBM, "The goal is to have computers start to interact in natural human terms across a range of applications and processes, understanding the questions that humans ask and providing answers that humans can understand and justify."IBM and Nuance Communications
Nuance Communications
Nuance Communications is a multinational computer software technology corporation, headquartered in Burlington, Massachusetts, USA, that provides speech and imaging applications...
Inc. are partnering for the research project to develop a commercial product during the next 18 to 24 months that will exploit Watson’s capabilities as a clinical decision support system
Clinical decision support system
Clinical decision support system is an interactive decision support system Computer Software, which is designed to assist physicians and other health professionals with decision making tasks, as determining diagnosis of patient data. A working definition has been proposed by Dr...
to aid the diagnosis and treatment of patients. Physicians at Columbia University
Columbia University
Columbia University in the City of New York is a private, Ivy League university in Manhattan, New York City. Columbia is the oldest institution of higher learning in the state of New York, the fifth oldest in the United States, and one of the country's nine Colonial Colleges founded before the...
are helping identify critical issues in the practice of medicine where the Watson technology may be able to contribute and physicians at the University of Maryland
University of Maryland
When the term "University of Maryland" is used without any qualification, it generally refers to the University of Maryland, College Park.University of Maryland may refer to the following:...
are working to identify the best way that a technology like Watson could interact with medical practitioners to provide the maximum assistance. It has been also suggested by Robert C. Weber, IBM's general counsel
General Counsel
A general counsel is the chief lawyer of a legal department, usually in a corporation or government department. The term is most used in the United States...
, that Watson may be used for legal research.
Watson is based on commercially available IBM Power 750 servers that have been marketed since February 2010. IBM also intends to market the DeepQA software to large corporations, with a price in the millions of dollars, reflecting the $1 million needed to acquire a server that meets the minimum system requirement to operate Watson. IBM expects the price to decrease substantially within a decade as the technology improves.
Commentator Rick Merritt said that "there's another really important reason why it is strategic for IBM to be seen very broadly by the American public as a company that can tackle tough computer problems. A big slice of Big Blue's pie comes from selling to the U.S. government some of the biggest, most expensive systems in the world."
Implications for artificial intelligence
Philosopher John SearleJohn Searle
John Rogers Searle is an American philosopher and currently the Slusser Professor of Philosophy at the University of California, Berkeley.-Biography:...
argues that Watson—despite impressive capabilities—could not actually think. Drawing on his Chinese room
Chinese room
The Chinese room is a thought experiment by John Searle, which first appeared in his paper "Minds, Brains, and Programs", published in Behavioral and Brain Sciences in 1980...
thought experiment, Searle claims that Watson, like other computational machines, is only capable of manipulating symbols, but has no ability to understand the meaning of those symbols; however, Searle's experiment may be flawed.
See also
- Deep Blue (chess computer)
- Blue GeneBlue GeneBlue Gene is a computer architecture project to produce several supercomputers, designed to reach operating speeds in the PFLOPS range, and currently reaching sustained speeds of nearly 500 TFLOPS . It is a cooperative project among IBM Blue Gene is a computer architecture project to produce...
- Artificial intelligenceArtificial intelligenceArtificial intelligence is the intelligence of machines and the branch of computer science that aims to create it. AI textbooks define the field as "the study and design of intelligent agents" where an intelligent agent is a system that perceives its environment and takes actions that maximize its...
- Commonsense knowledge problem
- Natural Language ProcessingNatural language processingNatural language processing is a field of computer science and linguistics concerned with the interactions between computers and human languages; it began as a branch of artificial intelligence....
- Strong AIStrong AIStrong AI is artificial intelligence that matches or exceeds human intelligence — the intelligence of a machine that can successfully perform any intellectual task that a human being can. It is a primary goal of artificial intelligence research and an important topic for science fiction writers and...
External links
- Watson homepage
- DeepQA homepage
- About Watson on Jeopardy.com
- Smartest Machine on Earth (PBS NOVA documentary about the making of Watson)
- POWER7
- Power Systems
- Noam Chomsky on Watson and AI
- The Watson Trivia Challenge. The New York TimesThe New York TimesThe New York Times is an American daily newspaper founded and continuously published in New York City since 1851. The New York Times has won 106 Pulitzer Prizes, the most of any news organization...
. June 16, 2010. - #IBMWatson TwitterTwitterTwitter is an online social networking and microblogging service that enables its users to send and read text-based posts of up to 140 characters, informally known as "tweets".Twitter was created in March 2006 by Jack Dorsey and launched that July...
hashtag
J! Archive
- Jeopardy! Show #6086 - Game 1, Part 1
- Jeopardy! Show #6087 - Game 1, Part 2
- Jeopardy! Show #6088 - Game 2
Videos
- PBS NOVA documentary on the making of Watson (3:59), IBM (2:29), IBMLabs (21:42), IBMLabs