Important lessons have emerged from the study of intelligence warning failures, but some common misconceptions have prevented the intelligence community from recognizing and incorporating these lessons. The first lesson is that analysis, rather than collection, is the most effective way to improve warning. The focus on improving warning normally turns to intelligence collection, rather than analysis.[13] That trend continued after the 11 September terrorist attacks.[14] Warning failures are rarely due to inadequate intelligence collection, but are more frequently due to erroneous analysis, and are most often due to decision makers ignoring intelligence,[15] largely because analytic product is weak.[16] Thus, the problem points mainly to analysis. Empirical research shows that more information does not improve the accuracy of analysts’ assessments; it merely improves analysts’ certainty in their assessments.[17] Additional research finds “an intensified collection effort does not necessarily lead to better analysis and more accurate estimates; when the additional information contains a large portion of noise, the risks of another intelligence failure leading to surprise may even increase.”[18] The evidence suggests there is more value in the available information than current analytical technique reveals. The second lesson is that hiring a smart person does not necessarily lead to good analysis. Many argue that “The best way to ensure high-quality analysis is to bring high-quality analysts into the process.”[19] Their reasoning is that many intelligent minds working together are bound to produce a good assessment. According to research, they are wrong.[20] Studies show that “frequently groups of smart, well-motivated people . . . agree . . . on the wrong solution. . . . They didn’t fail because they were stupid. They failed because they followed a poor process in arriving at their decisions.”[21] The third lesson is that a systematic process is the most effective way to facilitate good analysis. A combination of intuition and structured technique in a systematic process is better than intuition alone for intelligence analysis. “Doing something systematic is better in almost all cases than seat-of-the-pants prediction.”[22] This assertion is the subject of debate in the intelligence community.

The long-standing debate over whether structured techniques work on complex problems, such as intelligence analysis, divides many in the intelligence community.[23] Many analysts argue that structured techniques cannot account for the infinite number of variables involved in complex problems and that intuition is better.[24] The non-structured intuitive approach has become the norm in the intelligence community.[25] However, others point out that research shows intuitive judgments “seldom take proper account of all the information available”[26] and that systematic process can do better.[27] Both sides agree that intuition cannot be eliminated from intelligence analysis because a great deal of intelligence information is qualitative, or “information that can be captured that is not numerical in nature.”[28] Since non-numerical information cannot be counted, an analyst must assess its value subjectively using intuition. Structured technique proponents point out that researchers have identified many recurring shortcomings in intuitive judgments.[29] Since intuition cannot be eliminated from intelligence analysis, analysts must find ways to limit their vulnerability to the hidden danger of intuition. Structured techniques have been shown to guard against intuitive pitfalls, so some intelligence community professionals have proposed applying this technique in intelligence analysis. A key misunderstanding in the debate over intuition versus structured technique is that an analyst must choose one or the other.[30] In fact, both intuition and structured technique can be used together in a systematic process. “Anything that is qualitative can be assigned meaningful numerical values. These values can then be manipulated to help us achieve greater insight into the meaning of the data and to help us examine specific hypotheses.”[31]

To understand the debate over intuition versus structured technique, it is necessary to understand three key terms: intuition, structured technique, and systematic process. Intuition is “a feeling or instinct that does not use demonstrative reasoning processes and cannot be adequately explained by the analyst with the available evidence.”[32] The American Heritage Dictionary describes intuition as the “act or faculty of knowing or sensing without the use of rational process; immediate cognition.”[33] Dr. Gary Klein, a pioneer in applying intuitive decision-making theory to national security, defines intuition as a “recognitional, pattern-matching process that flows from experience.”[34] Cognitive psychologists assert that intuition is a “very rapid processing of information by the subconscious, relying on data already reviewed and knowledge already possessed.”[35] With intuitive analysis, the analyst cannot demonstrate his reasoning process. “In theory . . . an intuitive decision can take account of knowledge you possess but can’t put into words. And in principle your mind may be able to process information in a more complex and subtle way than you could formalize in a decision rule.”[36] “[J]udgment, insight and intuition. These methods are more abstract, so they are harder to teach. Since they are closer to an art than a science, they are also harder to understand.”[37] Research on how people with displayed intuition become experts found that successful chess masters, mathematicians, tennis players, aircraft pilots, and cab drivers all have experience in quickly perceiving large meaningful patterns.[38] “Research also suggests that an individual requires approximately 10-15 years of experience to become an expert in a field and exhibit intuitive decision-making capabilities.”[39] “[R]esearch indicates that maximizing the ability to make rapid analytical assessments is related less to choosing a person with special capabilities and more to creating and maintaining the motivation for long-term continued training.”[40]

Structured technique is a method used to “separate and logically organize the constituent elements of a problem [in order] to enhance analysis and decision-making.”[41] Science is used to develop rules in structured technique. According to the scientific rules, a topic is broken down, its multiple components judged, and then formed into an overarching assessment of the topic. Structured technique automates recurring judgments (according to the rules), so that they are completed with more speed and accuracy. “Since the process is rational, it can be taught to, and mastered by, almost any serious professional.”[42]

Systematic process is the combination of the structured technique(s) and intuition.[43] Intuition and science are used to make judgments at certain steps in the systematic process. This process limits intuitive judgments to small pieces of information. Intuition and science are used to determine which structured techniques work best on which topic. Intuition and science are used to develop rules on how to best combine structured techniques and intuition into an overall systematic process.

Many structured techniques have been developed for different needs in analysis. Some are very simple procedures, while others are complex mathematical formulas or multi-step processes. They all share the function of applying consistency in recurring judgments on different sets of data. Examples of structured techniques include lists, indicator filtering, and hypothesis testing. Lists are the simplest forms of structure. They assemble items into meaningful groups or sequences that help analysts to comprehend, judge, and act on data in an organized and meaningful way. Lists enable the mind to deal with larger amounts of data because lists group information into smaller, meaningful sections. Indicators are “those [collectable] things that would have to happen and those that would likely happen as [a] scenario unfolded.”[44] Thus, indicators provide the foundation for warning analysis. Indicator filtering is a technique that enables analysts to filter out incoming intelligence that does not meet the criteria of what analysts have judged to be important, i.e., the indicators. Since intelligence analysts are overwhelmed with information, indicator filtering is a technique that enables analysts to separate the signals from the noise. Computer programs (taxonomies) can help analysts with this process by using key words to automate indicator filtering. According to research, computer programs are just as accurate as humans in their analysis.[45] By combining both, analysts can increase accuracy and save time and manpower.

Hypothesis testing is the iconic example of structured technique in the intelligence community, and it is the fundamental practice of the modern scientific method.[46] In hypothesis testing, the analyst tracks evidence that is consistent and inconsistent with the potential hypotheses. In the process of hypothesis testing, the goal is to identify the correct hypothesis by eliminating or disproving each incorrect hypothesis with one piece of contradictory evidence rather than trying to prove a hypothesis by amassing supporting evidence. In controlled scientific experiments, where a piece of evidence can be known to be true with 100 percent certainty, only one piece of contradictory evidence is necessary to disprove a hypothesis. However, in the intelligence community realm, little information is known with absolute certainty, so one piece of contradictory evidence is not always enough to discount a hypothesis. When a piece of information is not known for certain to be true, as is often the case in the intelligence community, a person can assign a numerical level of confidence or probability percentage to represent the level of certainty. For example, analysts could represent their assessment of the validity of a piece of evidence on a five-level scale according to the following standardized language: Almost Certainly True (~90%), Probably True (~70%); Unknown (~50%)[47] ; Probably Not True (~30%); Almost Certainly Not True (~10%). This technique applies structure by using numerical values rather than just words, which can be misunderstood.

Analysts have a responsibility to improve their communication with decision makers. Richards Heuer, author of The Psychology of Intelligence Analysis[48] explains: “Verbal expressions of uncertainty — such as possible, probable, unlikely, may, and could — have long been recognized as sources of ambiguity and misunderstanding.”[49] Although the use of precise language may be recognized as valuable, the concept has not had much staying power. Intelligence community assessments are not required to give numbered estimates. The subject is still debated today when some analysts try to include numbered estimates.[50] Both words and numbers should be used in a standardized, systematic way. Decision makers have said they want to hear probabilities rather than wait to hear near certainties. As former Secretary of State Colin Powell explained, “What I need to do is operate when the probability of something happening is about, not .2, but about .7. I can’t wait until it’s .99 because by then it’s too late.”[51] Another benefit of using numbers rather than only words to represent assessments is that it enables analysts to combine the assessments according to a consistent quantitative rule set. This also enables the use of a computer to combine hundreds of small assessments into an overarching larger assessment.

Once numerical values have been assigned to pieces of evidence, analysts can combine data according to a consistent rule set that they judge appropriate for the data. Analysts can perform simple averaging or weighted ranking in averaging, such as giving one factor double the weight of another factor. Utility matrix logic is a simple technique that enables an analyst to visually show how two values of two factors cross reference in a matrix to form a value for a third factor. Bayesian theorem, also known as Bayes’s Theorem, is a rule set specifically designed for calculating conditional probabilities.[52] Analysts can create any other mathematical formula that will provide consistency and accuracy in repeatedly applying pre-determined intuitive choice(s). For example, if an expert makes an intuitive judgment that factor x should carry triple the weight of factor y, then the mathematical formula could consistently, accurately, and quickly apply that standard.[53]

Structured technique emerged in the scientific community long before it emerged in the intelligence community. Hypothesis testing is the fundamental practice of the modern scientific method and it is the iconic example of structured technique in the intelligence community. “The scientific method is not a set of rules . . . Rather it is the general procedure that scientists usually follow.”[54] Some efforts to apply the scientific method to a topic have not properly followed the principles of the scientific method. Those faulty efforts are called pseudoscience.[55] Pseudoscience is characterized by over reliance on confirming hypotheses rather than disproving them, lack of disclosure of data and methodology for testing by others, and the use of vague and untestable claims. Some people have asserted that the effort to apply the scientific method to intelligence analysis is pseudoscience because the variables involved in qualitative topics, such as intelligence analysis, cannot be fully quantified since they are often “so complex, countless, and incomplete.”[56]

Debate over structured technique was sparked in the intelligence community when Richards Heuer began conducting “methodological projects in political analyses” in the CIA in the latter half of the 1970s.[57] Between 1978 and 1986, he published his findings in a series of articles “for internal use within the CIA Directorate of Intelligence [DI],”[58] in which he explains that “the initial attitude . . . toward our unconventional [structured technique] proposals typically ranged from skepticism to hostility. Equally typical, however, has been their post-project appraisal that the work was interesting and well worth doing.”[59] Despite this, analysts did not embrace structured technique, citing numerous problems, including time constraints.[60]

In the mid-1990s, structured technique got significant attention again, this time from Congress and the media when policy makers, the media, and the public questioned the findings of the Rumsfeld Commission report on the threat of ballistic missiles. The 1998 report’s findings differed from the 1995 National Intelligence Estimate on the ballistic missile threat, and critics of the commission’s findings attacked its methodology. The Rumsfeld Commission used alternative hypothesis testing and in doing so reported not only on what they knew but also on what they did not know regarding alternative hypotheses.[61] That technique became a subject of controversy because of the concern that it could create an overly alarming assessment.[62]

Additionally, Douglas MacEachin, the CIA’s Deputy Director for Intelligence from 1993 to 1996, launched a “major new effort to raise analytical standards” in response to “pressures on CIA for better analytic performance in the wake of alleged ‘intelligence failures’ concerning Iraq’s invasion of Kuwait.”[63] MacEachin’s primary efforts to get structured technique implemented in analysis included new training requirements on structured techniques with a focus on “linchpin analysis” (another phrase for hypothesis testing).[64] Overall, as one professional from the CIA intelligence community documents, “In the mid-1990s, CIA focused attention on analytic tradecraft and produced a series of notes — accessible on the CIA’s website — which elaborate on some of the skills and methods used by [Directorate of Intelligence] intelligence analysts, such as addressing U.S. interests in DI assessments, access, and credibility to policy makers, articulation of assumptions, use of facts and sourcing, and effective use of unique intelligence.”[65] In 1999, Heuer’s internal CIA publications were made public in the book Psychology of Intelligence Analysis.[66] The book outlines a process based on hypothesis testing called “Analysis of Competing Hypotheses” (ACH), which is a “procedure [that] was developed by the author for use by intelligence analysts dealing with a set of particularly difficult problems.”[67] The debate over methodology has continued in intelligence literature as professionals on both sides have published their opinions.

Argument 1: Many analysts argue that structured techniques cannot account for the infinite number of variables involved in complex problems and that intuition can do better.[68] “Intelligence analysts who do use non-structured methods for qualitative analysis argue that structured methods too narrowly define a problem and ignore factors that cannot be measured. They assert that their intuitive approach produces superior results.”[69] This argument further contends that the application of the scientific method to intelligence analysis is pseudoscience because the variables involved in qualitative topics cannot be fully quantified.[70]

Argument 2: Even if studies show that structured techniques work on some qualitative problems, intelligence analysis involves fundamentally different qualitative problems, so those findings do not apply to intelligence analysis.[71] As James P. Finley explains, “Science can be of little help when dealing with the often irrational and unpredictable human mind [which intelligence analysts must assess].”[72]

Argument 3: Some analysts argue that structured techniques eliminate or reduce the use of intuition and therefore its valuable insights.[73] As one naval intelligence officer argues, “Overall, our rational methods are orderly and efficient, and they eliminate or reduce intangible factors like judgment, insight, and intuition.”[74]

Argument 4: Many people assert that they can “avoid the pitfalls of the mind, such as bias, by simply being aware of them.”[75] Within the intelligence community, “[m]any scholars suggest that analysts and decision makers alike must be more aware of the nature of judgmental biases and of the limitations of the intelligence process if they hope to reduce the incidence of surprise attack. This argument holds that if analysts and decision makers become aware of common perceptual errors, they may be able to avoid them. . . .”[76]

Argument 5: Many analysts argue that structured methods take too long.[77] Intelligence Officer Robert Gourley explains that “in a crisis decision-making environment . . . [there is] very little time to produce an assessment . . . [and there is] ongoing action so that proposed friendly decisions must be instantaneously evaluated and enemy reaction forecast.”[78] Intuitive proponents further argue that “the military’s use of precision, information-intensive weapons creates pressure on the intelligence analyst to deliver useful intelligence at faster rates. In the era of modern warfare, fast and precise weapons demand fast and precise intelligence. The intelligence analyst who is unable to conduct analysis quickly can create an information bottleneck in this environment.”[79]

Gary Klein, a cognitive psychologist and pioneer in applying intuition to national security decision-making, asserts, “These [structured technique] strategies sound good, but in practice they are often disappointing. They do not work under time pressure because they take too long. Even when there is enough time, they lack flexibility for handling rapidly changing field conditions.”[80] Richards Heuer, former head of the CIA’s methodology unit in the political analysis office, concedes that “the pressures of day-to-day duties, mail, and current intelligence reporting” can be a problem in applying structured techniques.[81] He acknowledges that structured techniques take time because analysts must complete two broad time consuming tasks, decomposition and externalization.[82] “Decomposition means breaking a problem down into its component parts. . . . Externalization means getting the decomposed problem out of one’s head and down on paper or on a computer screen in some simplified form that shows the main variables, parameters, or elements of the problem and how they relate to each other.”[83] Two other structured technique proponents (outside the intelligence community) and decision-making researchers, J. Edward Russo and Paul J.H. Schoemaker, acknowledge that “intuitive decision making does have at least one advantage. It certainly takes less time than making a decision with the systematic methods. . . . Everyone must decide many small questions intuitively.”[84] Former CIA analyst, Stephen Marrin, reports, “According to Boston University Professor — and 35-year intelligence veteran — Arthur Hulnick, ‘methodologists discovered that their new techniques were unpopular with analysts because they had no time to absorb these systems in the face of tight deadlines.’”[85]

The argument for structured technique is really an argument for combining intuition and structured technique in a systematic process (because intuition cannot be eliminated from intelligence analysis due to the qualitative information involved). This is the side of the argument that I support, and I provide some of my personal rationales for doing so throughout this section as an addition to the overwhelming evidence that already supports this view.

Research shows that intuitive judgments “seldom take proper account of all the information available”[86] and “[d]oing something systematic is better in almost all cases than seat-of-the-pants prediction.”[87] Moreover, as military intelligence analyst Robert D. Folker points out, “At first glance it seems that opponents of the scientific approach are criticizing the results of scientific methods. Yet they offer no empirical data to show which approach produces more accurate results, the scientific or the intuitive.”[88] Psychological experiments show that there are numerous reasons why intuitive judgments rarely account for all the available information. Individuals selectively remember or give weight to information based on the vividness and recentness of their exposure to it.[89] Information that is seen, heard, or experienced first-hand will have more impact on the mind than statistical data, which is impersonal and abstract but has substantially greater value as evidence.[90] People tend to ignore evidence that does not support their bias and interpret ambiguous information as confirming their bias.[91] When the mind is overwhelmed with information, the tendency to ignore information is magnified as part of a simplification technique to reduce the information down to a manageable size.[92] Furthermore, “intuitive judgments suffer from serious random inconsistencies due to fatigue, boredom, and all the factors that make us human.”[93]

Studies of medical experts have demonstrated the inconsistency in intuitive judgments. An experiment in which psychologists provided information from 96 X-rays to five radiologists found that “[o]n different days, the same expert [radiologist] will decide differently even a clear-cut question such as whether an x-ray indicates the presence of cancer.”[94] Moreover, “[p]eople making decisions by intuition alone usually suffer from ‘information overload’ and have a hard time applying simple rules consistently even when they try.”[95] When overwhelmed with information, the mind makes recurring, predictable kinds of mental errors — cognitive biases — as the mind attempts to simplify information that it is processing.[96] “People have difficulty keeping more than seven or so ‘chunks’ of information in mind at once.”[97] A study over 40 years ago documented in the article “The Magic Number Seven — Plus or Minus Two,” found that “seven — plus or minus two — is the number of things people can keep in their head all at once. That limitation on working memory is the source of many problems. People have difficulty grasping a problem in all its complexity. This is why we sometimes have trouble making up our minds. For example, we think first about the arguments in favor, and then about the arguments against, and we can't keep all those pros and cons in our head at the same time to get an overview of how they balance off against each other.”[98]

Many fields beside intelligence analysis have shown that being an expert does not create a magical intuition that enables a person to make judgments that are liberated from the common pitfalls of the mind as it processes large amounts of information.[99] “Indeed, the data shows that when experts fall victim to these traps, the effects can be aggravated by the confidence attached to expertise, both in their own view and in the perception of others.”[100] This explains why studies show that hiring smart people for their intuitive skills does not necessarily lead to good analysis, and that systematic process is a more significant factor in coming up with the right decision.[101] Studies show that “frequently groups of smart, well-motivated people . . . agree . . . on the wrong solution. . . . They didn’t fail because they were stupid. They failed because they followed a poor process in arriving at their decisions.”[102] For example, according to former Yale University psychologist Irving Janis, the planners involved in the disastrous invasion at the Bay of Pigs were “some of the smartest people in America. . . .”[103]

On the second point, more than 100 studies have shown that “[d]oing something systematic is better in almost all cases than seat-of-the-pants prediction.”[104] The evidence shows that “[y]ou can develop procedures that will make your decision better than your own unaided intuition.”[105] In one study, 130 University of Chicago Masters of Business Administration (MBA) students were asked to predict grade point averages of applicants intuitively by studying a summary sheet of factors about the applicant. The experiment controllers then conducted a regression analysis to determine on average what portion of their prediction was based on each factor. Using the data from the regression analysis, the experiment controllers created a new set of predictions using a structured technique called “bootstrap modeling,” which applied the averaged proportion weights of each factor in a systematic way. The result was that the bootstrap-modeling predictions were better than 81 percent of the 130 MBA students’ intuitive predictions. Thus, the MBA students could have beat their intuitive predictions if they had applied bootstrapping using their own judgments on how much to weigh each factor.[106] The bootstrapping technique has outperformed or equally performed to intuition alone in almost every study.[107]

There are two key errors in the argument that the application of the scientific method to qualitative intelligence analysis is pseudoscience due to the fact that the variables involved in qualitative topics cannot be fully quantified. First, in the scientific method, there is no requirement to quantify all the variables. The requirements are: “Stating a Problem . . . Collecting Data . . . through observation or measurement, or both . . . Forming Hypotheses . . . Testing Hypotheses . . . the scientist uses observation or, more often, experimentation . . . forming a Conclusion.”[108] Another basic expectation of the scientific method is to share all data and methodology so others can replicate the study, thereby allowing them to verify the results.[109] Secondly, other fields in which the scientific method has been acceptably applied, such as physics, biology, and medicine diagnostics, also involve “complex, countless, and incomplete variables,”[110] yet those applications are not considered pseudoscience. Furthermore, some may argue that the application of the scientific method to intelligence analysis is pseudoscience because statistical methods for testing hypotheses cannot be used.[111] However, that limitation does not invalidate the application of the scientific method to the social sciences. In the eighteenth and nineteenth centuries “the development of probability theory and statistics helped analysts analyze their observations and experiments” but they are not necessary.[112] The social sciences apply the scientific method differently than the physical sciences, but they still apply it in a way that is accepted as a legitimate application.[113] “Whenever possible, the scientist observes and measures things directly. . . . A physicist, for example, can measure atmospheric pressure with a barometer. A sociologist can gauge public opinion on a subject by asking people carefully worded questions.”[114] The same non- numerical application is thus acceptable for the intelligence field. As Heuer explains, “circumstances and insufficient data often preclude the application of rigorous scientific procedures in intelligence analysis — including, in particular, statistical methods for testing hypotheses. There is, however, certainly no reason why the basic conceptual strategy of looking for contrary evidence cannot be employed.”[115] According to scientific method principles, these kinds of applications are acceptable as long as the process is made transparent for peer review.[116]

Studies show that structured techniques work on qualitative topics that “deal with basic human mental processes and the results do seem consistent with personal experience in the Intelligence Community.”[117] Therefore, although intelligence analysis is a unique kind of qualitative topic, these studies would logically apply to any qualitative topic where analysts use basic human mental processes, including intelligence analysis. In 1999, one study focusing specifically on intelligence analysis “found that analysts who apply a structured method, hypothesis testing, in this case, to an intelligence problem, outperform those who rely on ‘analysis as-art,’ or the intuitive approach.”[118]

The human brain behaves the same in processing information regardless of whether the person is an intelligence analyst or not. Experimentation and study inside the intelligence community on cognitive biases are limited and are provided in large part by Richards Heuer. Heuer reports: “In similar kinds of psychological tests [regarding cognitive bias], in which experts, including intelligence analysts, were used as test subjects, the experts showed the same pattern of responses as students [non-intelligence test subjects]. . . . All this leads to the conclusion that the three biases are found in Intelligence Community personnel as well as in the specific test subjects.”[119] MacEachin sums it up: “This is not a phenomenon unique to intelligence; as Heuer's research demonstrates, it is part of the natural functioning of the human cognitive process, and it has been demonstrated across a broad range of fields ranging from medicine to stock market analysis.”[120]

Experimentation and study with the use of structured technique in the intelligence community are both limited and provided in large part by military intelligence analyst Robert D. Folker. Folker’s experiment compared the assessments of intelligence analysts who used the intuitive approach to intelligence analysts who used the structured technique hypothesis testing on two political–military scenarios.[121] The experiment was conducted at four of the nine Joint Intelligence Centers (JICs) at the US military’s Unified Commands.[122] At each JIC, an experimental group of 13 analysts were trained on the structured technique hypothesis testing. They and a control group of 13 analysts, who were not given the training, were given the same two scenarios. Each analyst individually assessed the scenarios using either the structured technique or intuition.[123] For both scenarios, the analysts who used structured technique had more correct answers than the analysts who used the intuitive approach.[124]

Heuer also experimented on structured technique in intelligence analysis. His studies show that “weaknesses and biases inherent in human thinking processes can be demonstrated through carefully designed experiments. They can be alleviated by conscious application of tools and techniques that should be in the analytical tradecraft toolkit of all intelligence analysts.”[125] The bottom line is that there is empirical evidence in the literature that the findings do apply to intelligence topics. Case studies and experiments show that “[s]ome structured methodologies . . . complement the analyst’s intuition and facilitate creativity,”[126] and can speed up the development of an analyst’s intuition.[127] An example of how structured technique complemented and sped up the development of analysts’ intuition was demonstrated by the Royal Dutch/Shell oil company. The company used a systematic process that involved obtaining numerical levels of confidence from analysts and providing regular feedback to analysts. Overconfidence in estimates, a pitfall that plagues intelligence analysts, was afflicting new geologists working for the company, costing it billions of dollars in drilling dry holes.[128] “When young geologists said there was a 50 percent chance of finding oil, oil would be found as little as 30 percent of the time.”[129] Although the new geologists had impeccable credentials and were made aware of each error, they continued to make the error of overconfidence for many years, until finally, “by the time geologists had many years of experience, Shell could trust that if they said wells on a dozen sites each had a 50 percent chance of producing oil, about half of the wells would produce.”[130] This on-the-job training was taking too long and costing too much money.[131] Shell then introduced a structured technique into a training program for new geologists. “Inexperienced geologists were shown summaries of past cases reflecting the myriad factors that might predict oil deposits. For each, they had to make a prediction with a numerical level of confidence. Then, they received feedback, they were told whether or not oil was actually found at each of the sites. The training worked wonderfully. Now, when even junior Shell geologists predict there is a 50 percent chance of producing oil at a site, the company gets a hit about five times out of ten.”[132] In the short training program, the new geologists gained the intuitive insight to make more accurate estimates that experienced geologists had taken years to develop.

Heuer makes some key points about the benefits of applying structured technique to intuitive judgments. He says these “mental tools . . . help analysts keep an open mind, question assumptions, see different perspectives, develop new ideas, and recognize when it is time to change their minds.”[133] He further explains that “[l]earning creative problem- solving techniques does not change an analyst’s native-born talents but helps an analyst achieve his or her full potential. Most people have the ability to be more innovative than they themselves realize.”[134]

Both Heuer and Folker describe several mental tools including thinking backwards, lists, analysis of competing hypotheses, and organized brainstorming. Thinking backwards structures an analyst’s thoughts to go in the reverse direction than normal. “Thinking backwards changes the focus from whether something might happen to how it might happen. Putting yourself into the future creates a different perspective that keeps you from getting anchored in the present. Analysts will often find, to their surprise, that they can construct a quite plausible scenario for an event they had previously thought unlikely. Thinking backwards is particularly helpful for events that have a low probability but very serious consequences should they occur. . . .”[135] “Lists such as [Benjamin] Franklin made are the simplest structures.”[136] The simple process of writing lists (of pros and cons, indicators, alternative hypotheses, etc.) has been shown to stimulate creativity. “One specialist in creativity has observed that ‘for the purpose of moving our minds, pencils can serve as crowbars — just by writing things down and making lists that stimulate new associations.’”[137] Lists stimulate creativity because they “let us get the obvious and habitual answers out of the way, so that we can add to the list by thinking of other ideas beyond those that came first to mind,”[138] and because “they exploit people’s tendency to be a bit compulsive — we want to keep adding to them.”[139] It is an accepted truth that analysts need to question their assumptions.[140] “Experience tells us that when analytical judgments turn out to be wrong, it usually was not because the information was wrong. It was because an analyst made one or more faulty assumptions that went unchallenged. The problem is that analysts cannot question everything, so where do they focus their attention? . . . One advantage of the competing hypotheses approach . . . is that it helps identify the linchpin assumptions that swing a conclusion in one direction or another.”[141] Helping analysts to recognize those linchpin assumptions that they need to question is a significant accomplishment because the mind is very resistant to changing assumptions/expectations,[142] “We tend to perceive what we expect to perceive,’’[143] and a change in an assumption will change the way a person processes new information, which opens an analyst’s mind to recognizing if the evidence supports another hypothesis.[144] The mind is so resistant to change that it also causes an individual to selectively see information that supports an expectation, which compounds the problem of an erroneous judgment. “Many experiments have been conducted that show the extraordinary extent to which the information obtained by an observer depends upon the observer’s own assumptions and preconceptions. . . . If an analyst cannot think of anything that would cause a change of mind, his or her mind-set may be so deeply entrenched that the analyst cannot see the conflicting evidence.”[145] Thus, the use of the structured technique analysis of competing hypotheses, can help analysts open their minds to new intuitive ideas by showing them the assumptions they should question. Analysis of competing hypotheses also increases objectivity by making the assumptions explicit to the reader of the analysis. “Analysts do not achieve objective analysis by avoiding preconceptions. . . . Objectivity is achieved by making basic assumptions and reasoning as explicit as possible so that they can be challenged by others and analysts can, themselves, examine their validity.”[146]

Structured technique expert and former CIA analyst Morgan D. Jones asserts that structured technique actually helps our intuition by opening people’s minds. “The fact is structuring one’s analysis is the quickest, surest path to opening the mind to alternatives.”[147] Being open to alternatives is important because “Failure to consider alternatives fully is the most common cause of flawed or incomplete analysis.”[148] As Folker explains, research shows that “[s]ome structured methodologies, such as organized brainstorming techniques, complement the analyst’s intuition and facilitate creativity.”[149]

Empirical research shows that “tactics can improve your success beyond what you can achieve simply by being aware of the dangers [mental pitfalls].”[150] As Heuer explains, “Cognitive biases are similar to optical illusions in that the error remains compelling even when one is fully aware of its nature. Awareness of the bias, by itself, does not produce a more accurate perception.”[151] The Defense Advanced Research Projects Agency conducted experiments in decision analysis to determine whether a person could overcome a cognitive bias by being made aware of it. Tests showed that the bias persisted.[152] That finding applies to intelligence as well. Heuer replicated the experiment with intelligence analysts and got the same results, asserting that his test “supports the validity of the previous findings.”[153]

Systematic feedback as a training technique also helped analysts overcome the problem of overconfidence in weather forecasting and in bank loan judgments.[154]

Believe it or not, such regular feedback (plus training) has given weather forecasters an enviable record of reliability. When weather forecasters make a forecast, they see the actual outcome within hours. Thus, they learn to make good forecasts. If an experienced weather person says there is a 60 percent chance of rain, rain falls 59 percent of the time. When a 90 percent chance of showers is predicted, it rains 87 percent of the time. By requesting levels of confidence in weather predictions, supervisors in the National Weather Service have made the feedback more meaningful and created enormous benefits for themselves, for weather forecasters who wanted to improve, and for users of weather forecasts.[155]

Methodology can in fact save time and maximize the use of time if an analyst makes data entry into a shared structured framework part of his daily routine, and even automates certain parts of the process for which the analyst has made pre-determined, intuitive choices. Moreover, the difference between crisis and non-crisis situations must be acknowledged in discussion of this argument.

It is true that structured technique is not appropriate in crisis and heat-of-battle situations. The majority of intelligence assessments, however, are not created in those situations. Many assessments are created and tracked over years, especially intelligence warning assessments, such as a North Korean attack, ballistic missile proliferation, and weapons of mass destruction proliferation. Granted, a topic being monitored long-term can eventually become a crisis situation. However, if the analyst had been using structured technique up to that point, at least his starting point in the crisis would be from an assessment that developed from a rational process that took more of the previous variables into account. Structured technique would provide a reference for pattern recognition that might aid the analyst in a moment when he must make a fast decision. Even in a crisis situation, there may be time to use structured technique if a system is already up and running on a computer, and if it has been designed in a way that allows easy entry of new variables to the framework, new data, and new value judgments.

Analysts still have time constraints in non-crisis situations from the daily requirements of meetings, current intelligence, briefings, and email.[156] If an analyst makes data entry into the structured framework part of his daily routine and automates certain parts of the process for which he has made pre-determined, intuitive choices, then methodology can in fact save time rather than consume it. It is true that extra time is required to initially implement a systematic process. This can be done up front and then save time in the future. It is an investment that the intelligence community needs to make.

The majority of intelligence analysis is not done during a crisis. As Folker explains, “[T]he majority of Klein’s research has been conducted with leaders in the armed forces and emergency service providers such as firemen, but not specifically members of the Intelligence Community. There has always been a clash of cultures between operators (commanders and other decision-makers whose focus is on accomplishing the mission at hand) and intelligence analysts. As intelligence analysts have attempted to bridge the gap between the two cultures, they have tended to adopt the same intuitive decision-making strategies commonly used by operators.”[157] There is a time and place for purely intuitive decisions, but it is not in intelligence warning analysis, which should be deliberate, well thought-out, and justified with an explainable reasoning process. Ultimately, the intelligence community needs to collectively recognize the nature of different situations and make a plan that maximizes the opportunities for structured technique whenever possible.

If the intelligence community accepts that a universal set of analytic standards must be implemented for the standing intelligence-warning topics, then the next step is to identify which techniques are appropriate. No single technique can be characterized as the most effective or relevant for intelligence analysis and certain ones are appropriate for different phases of the analytic process. Additionally, any given structured technique has both strengths and weaknesses. It is important to combine multiple techniques in order to counter these weaknesses. One technique may apply to multiple phases of the analytic process. Moreover, the type of problem may affect the technique. For example, an indicator list for terrorism would be different from an indicator list for a conventional nation-state attack. Experts may judge that indicator X should carry double the weight of indicator Y when assessing situation/enemy A, but vice versa when assessing situation/enemy B. Therefore, a formula used to combine multiple indicators into an overarching assessment would be adjusted for different situations or enemies. Different audiences or policy makers may have different preferences regarding the way they prefer to see and process information. For that reason, the use of multiple techniques is valuable in order to cover the variety of preferences, similar to the way teachers present a single lesson in multiple formats for each of the auditory, visual, and hands-on learner types.

The following is a proposed model of the intelligence-warning process. It includes seven broad phases shown in Table 2. Structured techniques that are appropriate to support each phase are identified in the table. Seven essential elements of this model facilitate the seven phases, as follows: a list of prioritized indicators and indicator question sets helps analysts identify the key elements of information required to forecast the selected topic; a list of prioritized collection requirements, derived from the indicators and indicator question sets, helps analysts publish an intelligence- collection plan; a terrorism master database helps analysts consolidate the information that was collected; hypothesis- matrix templates or web pages that are automatically populated by the master database help analysts sort information; a partly automated systematic process that combines intuitive and structured techniques, including hypothesis testing, indicator filtering, matrix logic, chronological sorting, and question set guidance, helps analysts draw conclusions; re- prioritized collection requirements in the intelligence collection plan help analysts focus collectors on intelligence gaps in order to refine and update conclusions; and warning-assessment templates and web pages that combine narratives and color-coded warning-level graphics help analysts communicate conclusions and provide warning in multiple standardized formats that cater to different preferences. The value of the seven essential elements can be understood in the context of an analogy to constructing a building (see Table 2). Also shown in Table 2 is a recommendation on which techniques are appropriate to facilitate each phase of the intelligence-warning process. This is not an all-inclusive recommendation; it is a starting point to explain the concept of how to select and apply structured techniques in intelligence warning.

A more detailed explanation of how these techniques would integrate into a comprehensive standardized warning methodology for terrorism (which identifies 86 indicators of terrorism) is available upon request. This methodology could be applied to any intelligence topic (not just terrorism) by simply changing the list of indicators. The methodology consists of 32 tasks within the seven phases, which apply 27 types of structured techniques. The 32 tasks are: 19 daily tasks, three monthly tasks, five annual tasks, and five as-required tasks. The 19 daily tasks can be completed in one day, because tasks have been automated wherever possible. Each task explanation indicates whether it is executed by a computer program, Raw Reporting Profiler(s), Indicator Specialist(s), Senior Warning Officer(s), or some combination of them working together. An analyst can override any automatically calculated value with justification for rejecting the computer program, which executes the pre-determined logic agreed upon by the experts.