Minimum Intelligent Signal Test
Encyclopedia
The Minimum Intelligent Signal Test, or MIST, is a variation of the Turing test
proposed by Chris McKinstry
in which only boolean (yes/no or true/false) answers may be given to questions. The purpose of such a test is to provide a quantitative statistical measure of humanness, which may subsequently be used to optimize the performance of artificial intelligence
systems intended to imitate human responses.
McKinstry gathered approximately 80,000 proposition
s that could be answered yes or no, e.g.:
He called these propositions Mindpixel
s.
These questions test both specific knowledge of aspects of culture, and basic facts about the meaning of various words and concepts. It could therefore be compared with the SAT
, intelligence testing and other controversial measures of mental ability. McKinstry's aim was not to distinguish between shades of intelligence but to identify whether a computer program could be considered intelligent at all.
According to McKinstry, a program able to do much better than chance on a large number of MIST questions would be judged to have some level of intelligence and understanding. For example, on a 20-question test, if a program were guessing the answers at random, it could be expected to score 10 correct on average. But the probability
of a program scoring 20 out of 20 correct by guesswork is only one in 220, i.e. one in 1,048,576; so if a program were able to sustain this level of performance over several independent trials, with no prior access to the propositions, it should be considered intelligent.
such as chatterbot
s, saying that his questions could "kill" AI programs by quickly exposing their weaknesses. He contrasted his approach, a series of direct questions assessing an AI's capabilities, to the Turing Test
and Loebner Prize
method of engaging an AI in undirected typed conversation.
Critics of the MIST have noted that it would be easy to "kill" a McKinstry-style AI too, due to the impossibility of supplying it with correct answers to all possible yes/no questions by ways of a finite set of human-generated Mindpixels: the fact that an AI can answer the question "Is the sun bigger than my foot?" correctly does not mean that it can answer variations like "Is the sun bigger than (my hand | my liver | an egg yolk | Alpha Centauri A | ...)" correctly, too.
However, the late McKinstry might have replied that a truly intelligent, knowledgeable entity (on a par with humans) would be able to work out answers such as (yes | yes | yes | don't know | ...) by applying its knowledge of the relative sizes of the objects named. In other words, the MIST was intended as a test of AI, not as a suggestion for implementing AI.
It can also be argued that the MIST is a more objective test of intelligence than the Turing test, a subjective assessment that some might consider to be more a measure of the interrogator's gullibility than of the machine's intelligence. According to this argument, a human's judgment of a Turing test is vulnerable to the ELIZA effect
, a tendency to mistake superficial signs of intelligence for the real thing, anthropomorphizing the program. The response, suggested by Alan Turing
's essay "Can Machines Think?", is that if a program is a convincing imitation of an intelligent being, it is in fact intelligent. The dispute is thus over what it means for a program to have "real" intelligence, and by what signs it can be detected.
A similar debate exists in the controversy over Great Ape language
, in which nonhuman primates are said to have learned some aspects of sign languages but the significance of this learning is disputed.
Turing test
The Turing test is a test of a machine's ability to exhibit intelligent behaviour. In Turing's original illustrative example, a human judge engages in a natural language conversation with a human and a machine designed to generate performance indistinguishable from that of a human being. All...
proposed by Chris McKinstry
Chris McKinstry
Kenneth Christopher McKinstry was a researcher in artificial intelligence. He led the development of the MISTIC project which was launched in May 1996. He founded the Mindpixel project in July 2000, and closed it in December 2005...
in which only boolean (yes/no or true/false) answers may be given to questions. The purpose of such a test is to provide a quantitative statistical measure of humanness, which may subsequently be used to optimize the performance of artificial intelligence
Artificial intelligence
Artificial intelligence is the intelligence of machines and the branch of computer science that aims to create it. AI textbooks define the field as "the study and design of intelligent agents" where an intelligent agent is a system that perceives its environment and takes actions that maximize its...
systems intended to imitate human responses.
McKinstry gathered approximately 80,000 proposition
Proposition
In logic and philosophy, the term proposition refers to either the "content" or "meaning" of a meaningful declarative sentence or the pattern of symbols, marks, or sounds that make up a meaningful declarative sentence...
s that could be answered yes or no, e.g.:
- Is Earth a planet?
- Was Abraham Lincoln once President of the United States?
- Is the sun bigger than my foot?
- Do people sometimes lie?
He called these propositions Mindpixel
Mindpixel
Mindpixel was a web-based collaborative artificial intelligence project which aimed to create a knowledgebase of millions of human validated true/false statements, or probabilistic propositions. It ran from 2000 to 2005.-Description:...
s.
These questions test both specific knowledge of aspects of culture, and basic facts about the meaning of various words and concepts. It could therefore be compared with the SAT
SAT
The SAT Reasoning Test is a standardized test for college admissions in the United States. The SAT is owned, published, and developed by the College Board, a nonprofit organization in the United States. It was formerly developed, published, and scored by the Educational Testing Service which still...
, intelligence testing and other controversial measures of mental ability. McKinstry's aim was not to distinguish between shades of intelligence but to identify whether a computer program could be considered intelligent at all.
According to McKinstry, a program able to do much better than chance on a large number of MIST questions would be judged to have some level of intelligence and understanding. For example, on a 20-question test, if a program were guessing the answers at random, it could be expected to score 10 correct on average. But the probability
Probability
Probability is ordinarily used to describe an attitude of mind towards some proposition of whose truth we arenot certain. The proposition of interest is usually of the form "Will a specific event occur?" The attitude of mind is of the form "How certain are we that the event will occur?" The...
of a program scoring 20 out of 20 correct by guesswork is only one in 220, i.e. one in 1,048,576; so if a program were able to sustain this level of performance over several independent trials, with no prior access to the propositions, it should be considered intelligent.
Discussion
McKinstry criticized existing approaches to artificial intelligenceArtificial intelligence
Artificial intelligence is the intelligence of machines and the branch of computer science that aims to create it. AI textbooks define the field as "the study and design of intelligent agents" where an intelligent agent is a system that perceives its environment and takes actions that maximize its...
such as chatterbot
Chatterbot
A chatter robot, chatterbot, chatbot, or chat bot is a computer program designed to simulate an intelligent conversation with one or more human users via auditory or textual methods, primarily for engaging in small talk. The primary aim of such simulation has been to fool the user into thinking...
s, saying that his questions could "kill" AI programs by quickly exposing their weaknesses. He contrasted his approach, a series of direct questions assessing an AI's capabilities, to the Turing Test
Turing test
The Turing test is a test of a machine's ability to exhibit intelligent behaviour. In Turing's original illustrative example, a human judge engages in a natural language conversation with a human and a machine designed to generate performance indistinguishable from that of a human being. All...
and Loebner Prize
Loebner prize
The Loebner Prize is an annual competition in artificial intelligence that awards prizes to the chatterbot considered by the judges to be the most human-like. The format of the competition is that of a standard Turing test. In each round, a human judge simultaneously holds textual conversations...
method of engaging an AI in undirected typed conversation.
Critics of the MIST have noted that it would be easy to "kill" a McKinstry-style AI too, due to the impossibility of supplying it with correct answers to all possible yes/no questions by ways of a finite set of human-generated Mindpixels: the fact that an AI can answer the question "Is the sun bigger than my foot?" correctly does not mean that it can answer variations like "Is the sun bigger than (my hand | my liver | an egg yolk | Alpha Centauri A | ...)" correctly, too.
However, the late McKinstry might have replied that a truly intelligent, knowledgeable entity (on a par with humans) would be able to work out answers such as (yes | yes | yes | don't know | ...) by applying its knowledge of the relative sizes of the objects named. In other words, the MIST was intended as a test of AI, not as a suggestion for implementing AI.
It can also be argued that the MIST is a more objective test of intelligence than the Turing test, a subjective assessment that some might consider to be more a measure of the interrogator's gullibility than of the machine's intelligence. According to this argument, a human's judgment of a Turing test is vulnerable to the ELIZA effect
ELIZA effect
The ELIZA effect, in computer science, is the tendency to unconsciously assume computer behaviors are analogous to human behaviors.In its specific form, the ELIZA effect refers only to "the susceptibility of people to read far more understanding than is warranted into strings of symbols —...
, a tendency to mistake superficial signs of intelligence for the real thing, anthropomorphizing the program. The response, suggested by Alan Turing
Alan Turing
Alan Mathison Turing, OBE, FRS , was an English mathematician, logician, cryptanalyst, and computer scientist. He was highly influential in the development of computer science, providing a formalisation of the concepts of "algorithm" and "computation" with the Turing machine, which played a...
's essay "Can Machines Think?", is that if a program is a convincing imitation of an intelligent being, it is in fact intelligent. The dispute is thus over what it means for a program to have "real" intelligence, and by what signs it can be detected.
A similar debate exists in the controversy over Great Ape language
Great Ape language
Research into non-human great ape language has involved teaching chimpanzees, bonobos, gorillas, and orangutans to communicate with human beings and with each other using sign language, physical tokens, and lexigrams; see Yerkish...
, in which nonhuman primates are said to have learned some aspects of sign languages but the significance of this learning is disputed.
External links
- Minimum Intelligent Signal Test: An Alternative Turing Test, Chris McKinstry, Canadian Artificial Intelligence #41
- Gac-80k
- Open source software for performing the Minimum Intelligent Signal Test