Do you think like an intelligence analyst?

We live in an era of immense uncertainty. Tensions between great powers are rising, authoritarianism is spreading, while democracy declines and ongoing wars could broaden into wider conflicts.

To make informed decisions in turbulent times, governments rely on their intelligence agencies to address a wide range of questions often fraught with uncertainty, incomplete evidence, complex arguments and subtle inferences.  

Having enough people with the right skills and training is essential. Intelligence analysts need to be rigorous, insightful, imaginative and able to make nuanced assessments based on incomplete evidence. 

In an interview I recorded with Andrew Shearer, the Director General of National Intelligence, he said that they need people who are both “perfectionists in the sense that they don’t want to miss the one vital clue” but are also “comfortable with ambiguity” and able to operate “usefully within the bounds of that doubt”.  

This is of course, very challenging, even for the people carefully selected to work in intelligence agencies. 

Some national security experts think that Australia is “not generating enough workers with the necessary skills”. 

So we have devised a survey to understand why applications may be falling and what skills people think they need to apply (more on how you can participate later).

Sciences & Technology

Making big sense of big data: The quest to improve human reasoning

Constructing complex arguments and evaluating sources

After the Weapons of Mass Destruction Commission in the US, major reforms have taken place to improve the rigour of analytic assessments. 

The US intelligence community now has analytic tradecraft standards that its assessments should adhere to. 

A sample of their assessments is evaluated each year to check how well their agencies do on these standards. These internal evaluations found that “clear and logical argumentation” is a “difficult standard to meet” and is one with the most room for improvement. 

However, this is not a criticism of the analysts. For one thing, the data isn’t made public and no comparison has been made with other groups of society. 

I would guess that the argumentation skills of experienced intelligence analysts would be well above average, seeing that they spend much of their working day constructing complex arguments and evaluating sources. 

Types of reasoning used by analysts 

But more importantly, you don’t need to read too many declassified or unclassified intelligence assessments to realise how difficult it is to answer many of the questions intelligence analysts are required to address. 

For example, analysts frequently need to compare explanations for circumstantial evidence, as was the case when estimating how likely it was that Osama bin Laden was hiding in the Abbottabad compound and not someone else, like a criminal on the run. 

This abductive reasoning is commonplace but poorly understood. 

Or what is the best approach to approximate the strategic reasoning of state actors, as we need to when considering Xi Jinping’s intentions regarding Taiwan? 

Often, the literature focuses on common fallacies used in these types of arguments but does not provide prescriptive theories on how to do it well. 

Sciences & Technology

Another intelligence failure shows we need more rigour

For example, many people will have heard that a correlation by itself does not imply causation, but far fewer people understand what sort of argument is needed to support a causal claim.

Furthermore, while logical reasoning and critical thinking – which are “central” to the job of intelligence analysis – are skills where the right sort of training can be extremely beneficial, it is also difficult and time-consuming. 

Research my colleagues and I conducted for the US Intelligence Advanced Research Projects Activity (IARPA) suggests that a technique called computer-assisted argument mapping is one of the most effective ways of improving critical thinking skills in university students, and works just as well for intelligence analysts. 

And while I teach argument mapping several times a year in Canberra, this is just a drop in the ocean toward providing analysts in our National Intelligence Community with this state-of-the-art training.   

Recruiting analysts

From a national security perspective, it would probably be better if we had more Australians applying for roles in our National Intelligence Community, because a greater pool of applicants to draw from would make it easier to get the right mix of skills, experience and perspectives. 

More people are needed for the expanding forms of intelligence, like cyber-intelligence and Open Source Intelligence (OSINT). 

But at the same time, Human Intelligence (HUMINT) is as important as it has ever been, along with the people skills required to manage agents and interpret their information.  

In Australia, there is some concern about recruitment rates at the agencies that make up the National Intelligence Community (NIC). 

Education

Maybe the problem isn’t critical thinking, but how we assess it

The terms of reference for the Australian Government's 2024 Independent Intelligence Review stated that it would consider “Whether workforce decisions by the NIC at both the agency and community levels reflect a sufficiently strategic response to current and future workforce challenges … and offer options if recruitment targets cannot be met”. 

Ten of the sixty-three recommendations of the publicly released version of the report concerned “collective action on people and skills”, which focused on how to improve recruitment. 

People might not apply because they don’t know what skills intelligence agencies are looking for, because job ads refer to sensitive skills and knowledge that can’t be explained or only in very vague and generic terms.   

Intelligence agencies seem to agree with this and are starting to explain their mission more to students and the public.  

The Director-General of National Intelligence provides some good insights into the nature of the job, and he makes the point that they are looking for people with a wide range of skills and experience. 

However, it would be good to know more about what people think working as an intelligence analyst is like, how well they understand analytical reasoning and what factors people consider when contemplating a career in intelligence analysis.

So, my colleagues and I have designed a survey to help with these two related questions:

1. How do people go about evaluating analytic arguments? 

2. What do people think intelligence analysts do? 

A better understanding of how people evaluate analytical reasoning will help develop more efficient ways of teaching critical thinking skills.

The intelligence review noted the importance of shared foundation training that helps “generate common standards and platforms for analytical tradecraft”, and this research, like previous research done at the University of Melbourne, could be used to inform training on understanding the common standards for evaluating analytic reasoning. 

Sciences & Technology

Crowdsourcing security intelligence

If you would like to help out with this research, the survey is for Australians over 18 years old and takes about 90 minutes to complete.

It will be followed up with a webinar that will explain the results of the research and cover a couple of the Structured Analytic Techniques I’ve developed for the Intelligence Community.  

You never know, you might pick up some of the “analytic tradecraft” that Andrew Shearer says his analysts use “to pierce the fog of uncertainty and help illuminate at least the outlines of what’s really happening”.

To participate in the survey, please visit this link.