Analysis of Competing Hypotheses
Encyclopedia
The Analysis of Competing Hypotheses (ACH) provides an unbiased methodology
for evaluating multiple competing hypotheses for observed data. It was developed by Richards (Dick) J. Heuer, Jr.
, a 45-year veteran of the Central Intelligence Agency
, in the 1970s for use by the Agency and is used by analysts in various fields who make judgments that entail a high risk of error in reasoning. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.
ACH was indeed a step forward in intelligence analysis methodology, but it was first described in relatively informal terms. Producing the best available information from uncertain data remains the goal of both researchers, tool-builders, and industrial, academic, and government data analysts. Their domains include data mining
, cognitive psychology
and visualization, probability
and statistics
, etc. Abductive reasoning
is an earlier concept with points of similarity to ACH.
es, though there is a lack of strong empirical evidence to support this belief. Since the ACH requires the analyst to construct a matrix, the evidence and hypotheses can be backtracked
. This allows the decisionmaker or other analysts to see the sequence of rules and data that led to the conclusion.
Especially in intelligence, both governmental and business, analysts must always be aware that the opponent(s) is intelligent and may be generating information intended to deceive
. Since deception often is the result of a cognitive trap, Elsaesser and Stech use state-based hierarchical plan recognition (see abductive reasoning
) to generate causal explanations of observations. The resulting hypotheses are converted to a dynamic Bayesian network and value of information
analysis is employed to isolate assumptions implicit in the evaluation of paths in, or conclusions of, particular hypotheses. As evidence in the form of observations of states or assumptions is observed, they can become the subject of separate validation. Should an assumption or necessary state be negated, hypotheses depending on it are rejected. This is a form of root cause analysis
.
Evidence also presents a problem if it is unreliable. The evidence used in the matrix is static and therefore it can be a snapshot in time.
van Gelder has made the following criticisms:
For example, two tested hypotheses could be that Iraq has WMD or Iraq does not have WMD. If the evidence showed that it is more likely there are WMDs in Iraq then two new hypotheses could be formulated: WMD are in Baghdad or WMD are in Mosul. Or perhaps, the analyst may need to know what type of WMD Iraq has; the new hypotheses could be that Iraq has biological WMD, Iraq has chemical WMD and Iraq has nuclear WMD. By giving the ACH structure, the analyst is able to give a nuanced estimate.
.
Other work focuses less on probabilistic methods and more on cognitive and visualization extensions to ACH, as discussed by Madsen and Hicks. DECIDE, discussed under automation is visualization-oriented.
Work by Pope and Jøsang uses subjective logic
, a formal mathematical methodology that explicitly deals with uncertainty. This methodology forms the basis of the Sheba technology that is used in Veriluma's intelligence assessment software.
PARC ACH 2.0 was developed by Palo Alto Research Center (PARC) in collaboration with Richards J. Heuer, Jr. It is a standard ACH program that allows analysts to enter evidence and rate its credibility and relevance. Another useful program is the Decision Command software created by Dr. Willard Zangwill.
SSS Research, Inc. is an analytic research firm that created DECIDE. DECIDE not only allows analysts to manipulate ACH, but it provides multiple visualization products.
Competing Hypotheses is an open source
ACH implementation.
Methodology
Methodology is generally a guideline for solving a problem, with specificcomponents such as phases, tasks, methods, techniques and tools . It can be defined also as follows:...
for evaluating multiple competing hypotheses for observed data. It was developed by Richards (Dick) J. Heuer, Jr.
Richards Heuer
Richards J. Heuer, Jr. is a former CIA veteran of forty-five years and most known for his work on Analysis of Competing Hypotheses and his book, Psychology of Intelligence Analysis. The former provides a methodology for overcoming intelligence biases while the latter outlines how mental models...
, a 45-year veteran of the Central Intelligence Agency
Central Intelligence Agency
The Central Intelligence Agency is a civilian intelligence agency of the United States government. It is an executive agency and reports directly to the Director of National Intelligence, responsible for providing national security intelligence assessment to senior United States policymakers...
, in the 1970s for use by the Agency and is used by analysts in various fields who make judgments that entail a high risk of error in reasoning. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.
ACH was indeed a step forward in intelligence analysis methodology, but it was first described in relatively informal terms. Producing the best available information from uncertain data remains the goal of both researchers, tool-builders, and industrial, academic, and government data analysts. Their domains include data mining
Data mining
Data mining , a relatively young and interdisciplinary field of computer science is the process of discovering new patterns from large data sets involving methods at the intersection of artificial intelligence, machine learning, statistics and database systems...
, cognitive psychology
Cognitive psychology
Cognitive psychology is a subdiscipline of psychology exploring internal mental processes.It is the study of how people perceive, remember, think, speak, and solve problems.Cognitive psychology differs from previous psychological approaches in two key ways....
and visualization, probability
Probability
Probability is ordinarily used to describe an attitude of mind towards some proposition of whose truth we arenot certain. The proposition of interest is usually of the form "Will a specific event occur?" The attitude of mind is of the form "How certain are we that the event will occur?" The...
and statistics
Statistics
Statistics is the study of the collection, organization, analysis, and interpretation of data. It deals with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments....
, etc. Abductive reasoning
Abductive reasoning
Abduction is a kind of logical inference described by Charles Sanders Peirce as "guessing". The term refers to the process of arriving at an explanatory hypothesis. Peirce said that to abduce a hypothetical explanation a from an observed surprising circumstance b is to surmise that a may be true...
is an earlier concept with points of similarity to ACH.
Process
Heuer outlines the ACH process in considerable depth in his book, Psychology of Intelligence Analysis. It consists of the steps:- Hypothesis – The first step of the process is to identify all potential hypotheses, preferably using a group of analysts with different perspectives to brainstormBrainstormingBrainstorming is a group creativity technique by which a group tries to find a solution for a specific problem by gathering a list of ideas spontaneously contributed by its members...
the possibilities. The process discourages the analyst from choosing one "likely" hypothesis and using evidence to prove its accuracy. Cognitive biasCognitive biasA cognitive bias is a pattern of deviation in judgment that occurs in particular situations. Implicit in the concept of a "pattern of deviation" is a standard of comparison; this may be the judgment of people outside those particular situations, or may be a set of independently verifiable...
is minimized when all possible hypotheses are considered. - Evidence – The analyst then lists evidence and arguments (including assumptions and logical deductions) for and against each hypothesis.
- Diagnostics – With the use of a matrixMatrix (mathematics)In mathematics, a matrix is a rectangular array of numbers, symbols, or expressions. The individual items in a matrix are called its elements or entries. An example of a matrix with six elements isMatrices of the same size can be added or subtracted element by element...
, the analyst applies evidence against each hypothesis in an attempt to disprove as many theories as possible. Some evidence will have greater "diagnosticity" than others—that is, some will be most helpful in judging the relative likelihood of alternative hypotheses. According to Heuer, this step is the most important. Instead of looking at one hypothesis and all evidence, the matrix forces the analyst to consider one piece of evidence and examine it against all possible hypotheses. - Refinement – At this point the analyst reviews his/her findings and then identifies gaps and collects the additional evidence needed to refute as many of the remaining hypotheses as possible.
- Inconsistency – The analyst seeks to draw tentative conclusions about the relative likelihood of each hypothesis, with less consistency equaling less likelihood. The least consistent hypotheses are eliminated. While the matrix generates a mathematical sum, it is up to the analyst to use his/her judgment to decide on the conclusion.
- Sensitivity – The analyst then tests his/her conclusions using sensitivity analysisSensitivity analysisSensitivity analysis is the study of how the variation in the output of a statistical model can be attributed to different variations in the inputs of the model. Put another way, it is a technique for systematically changing variables in a model to determine the effects of such changes.In any...
, which weighs how the conclusion would be affected if key evidence or arguments were wrong, misleading, or subject to a different interpretation. The validity of key evidence and the consistency of important arguments are double-checked to assure that the conclusion's linchpins and drivers are sound. - Conclusions and evaluation – The analyst then provides the decisionmaker with his/her conclusions, as well as a summary of alternatives that were considered and why they were rejected. The analyst also identifies milestones in the process that can serve as indicators in future analyses.
Strengths
There are many benefits of doing an ACH matrix. It is auditable. It is widely believed to help overcome cognitive biasCognitive bias
A cognitive bias is a pattern of deviation in judgment that occurs in particular situations. Implicit in the concept of a "pattern of deviation" is a standard of comparison; this may be the judgment of people outside those particular situations, or may be a set of independently verifiable...
es, though there is a lack of strong empirical evidence to support this belief. Since the ACH requires the analyst to construct a matrix, the evidence and hypotheses can be backtracked
Backtracking
Backtracking is a general algorithm for finding all solutions to some computational problem, that incrementally builds candidates to the solutions, and abandons each partial candidate c as soon as it determines that c cannot possibly be completed to a valid solution.The classic textbook example...
. This allows the decisionmaker or other analysts to see the sequence of rules and data that led to the conclusion.
Weaknesses
The process to create an ACH is time consuming. The ACH matrix can be problematic when analyzing a complex project. It can be cumbersome for an analyst to manage a large database with multiple pieces of evidence.Especially in intelligence, both governmental and business, analysts must always be aware that the opponent(s) is intelligent and may be generating information intended to deceive
Deception
Deception, beguilement, deceit, bluff, mystification, bad faith, and subterfuge are acts to propagate beliefs that are not true, or not the whole truth . Deception can involve dissimulation, propaganda, and sleight of hand. It can employ distraction, camouflage or concealment...
. Since deception often is the result of a cognitive trap, Elsaesser and Stech use state-based hierarchical plan recognition (see abductive reasoning
Abductive reasoning
Abduction is a kind of logical inference described by Charles Sanders Peirce as "guessing". The term refers to the process of arriving at an explanatory hypothesis. Peirce said that to abduce a hypothetical explanation a from an observed surprising circumstance b is to surmise that a may be true...
) to generate causal explanations of observations. The resulting hypotheses are converted to a dynamic Bayesian network and value of information
Value of information
Value of information is the amount a decision maker would be willing to pay for information prior to making a decision.-Similar terms:...
analysis is employed to isolate assumptions implicit in the evaluation of paths in, or conclusions of, particular hypotheses. As evidence in the form of observations of states or assumptions is observed, they can become the subject of separate validation. Should an assumption or necessary state be negated, hypotheses depending on it are rejected. This is a form of root cause analysis
Root cause analysis
Root cause analysis is a class of problem solving methods aimed at identifying the root causes of problems or events.Root Cause Analysis is any structured approach to identifying the factors that resulted in the nature, the magnitude, the location, and the timing of the harmful outcomes of one...
.
Evidence also presents a problem if it is unreliable. The evidence used in the matrix is static and therefore it can be a snapshot in time.
van Gelder has made the following criticisms:
- ACH demands that the analyst makes too many discrete judgments, a great many of which contribute little if anything to discerning the best hypothesis
- ACH misconceives the nature of the relationship between items of evidence and hypotheses by supposing that items of evidence are, on their own, consistent or inconsistent with hypotheses.
- ACH treats the hypothesis set as "flat", i.e. a mere list, and so is unable to relate evidence to hypotheses at the appropriate levels of abstraction
- ACH cannot represent subordinate argumentation, i.e. the argumentation bearing up on a piece of evidence.
- ACH activities at realistic scales leave analysts disoriented or discombobulated.
Structured Analysis of Competing Hypotheses
The Structured Analysis of Competing Hypotheses offers analysts an improvement over the limitations of the original ACH. The SACH maximizes the possible hypotheses by allowing the analyst to split one hypothesis into two complex ones.For example, two tested hypotheses could be that Iraq has WMD or Iraq does not have WMD. If the evidence showed that it is more likely there are WMDs in Iraq then two new hypotheses could be formulated: WMD are in Baghdad or WMD are in Mosul. Or perhaps, the analyst may need to know what type of WMD Iraq has; the new hypotheses could be that Iraq has biological WMD, Iraq has chemical WMD and Iraq has nuclear WMD. By giving the ACH structure, the analyst is able to give a nuanced estimate.
Other approaches to formalism
One method, by Valtorta and colleagues uses probabilistic methods, adds Bayesian analysis to ACH. The work by Akram and Wang applies paradigms from graph theoryGraph theory
In mathematics and computer science, graph theory is the study of graphs, mathematical structures used to model pairwise relations between objects from a certain collection. A "graph" in this context refers to a collection of vertices or 'nodes' and a collection of edges that connect pairs of...
.
Other work focuses less on probabilistic methods and more on cognitive and visualization extensions to ACH, as discussed by Madsen and Hicks. DECIDE, discussed under automation is visualization-oriented.
Work by Pope and Jøsang uses subjective logic
Subjective logic
Subjective logic is a type of probabilistic logic that explicitly takes uncertainty and belief ownership into account. In general, subjective logic is suitable for modeling and analysing situations involving uncertainty and incomplete knowledge...
, a formal mathematical methodology that explicitly deals with uncertainty. This methodology forms the basis of the Sheba technology that is used in Veriluma's intelligence assessment software.
Automation
A few online and downloadable tools help automate the ACH process. These programs leave a visual trail of evidence and allow the analyst to weigh evidence.PARC ACH 2.0 was developed by Palo Alto Research Center (PARC) in collaboration with Richards J. Heuer, Jr. It is a standard ACH program that allows analysts to enter evidence and rate its credibility and relevance. Another useful program is the Decision Command software created by Dr. Willard Zangwill.
SSS Research, Inc. is an analytic research firm that created DECIDE. DECIDE not only allows analysts to manipulate ACH, but it provides multiple visualization products.
Competing Hypotheses is an open source
Open source
The term open source describes practices in production and development that promote access to the end product's source materials. Some consider open source a philosophy, others consider it a pragmatic methodology...
ACH implementation.
See also
- Intelligence analysisIntelligence analysisIntelligence analysis is the process of taking known information about situations and entities of strategic, operational, or tactical importance, characterizing the known, and, with appropriate statements of probability, the future actions in those situations and by those entities...
- Intelligence (information gathering)Intelligence (information gathering)Intelligence assessment is the development of forecasts of behaviour or recommended courses of action to the leadership of an organization, based on a wide range of available information sources both overt and covert. Assessments are developed in response to requirements declared by the leadership...
- Cognitive biasCognitive biasA cognitive bias is a pattern of deviation in judgment that occurs in particular situations. Implicit in the concept of a "pattern of deviation" is a standard of comparison; this may be the judgment of people outside those particular situations, or may be a set of independently verifiable...
- Words of Estimative ProbabilityWords of Estimative ProbabilityWords of Estimative Probability are terms used by intelligence analysts in the production of analytic reports to convey the likelihood of a future event occurring. They express the extent of their confidence in the finding. A well-chosen WEP provides a decision maker with an unambiguous estimate...