ACH, Cognitive Bias, and Politics

On Friday I got wiped out by the ‘flu, to the extent that I spent much of the last five days sleeping, or trying to. Fun, fun.

Somewhere along the way, though, I read an interesting paper on ACH – the Analysis of Competing Hypotheses, which I’ve probably written about at some point in the past. Specifically, it’s a thesis paper by Drew Brasfield, a (former) student of Mercyhurst professor Kristan Wheaton, which attempted to determine both how objectively effective ACH is as an analytic tool, and how effective it is at mitigating cognitive bias.

Why should you care? ACH is a relatively simple analytic tool, and is also fairly popular, probably for that very reason. It’s fairly easy to make use of, you don’t need to be a math whiz or computer genius to use it, and no special tools are required. It’s also just about understandable to laypeople with a very brief explanation, so it’s (declared) use might under some circumstances engender (extra) confidence in results derived from it, because there’s nothing especially mysterious or complicated about it. (Whereas if someone asks how you arrived at a conclusion and your explanation is riddled with jargon and scary-sounding words like “gaussian”, it’s easy to suspect you’re blowing smoke up their bum.)

But does it actually work?

That was the question Brasfield set out to answer.

Short answer: Maybe. The magic eight-ball says (statistically) “inconclusive”, try again later.

With a small sample size of around fifty people, Brasfield’s test subjects did nine percent better on what was basically an either-or problem (a domestic political election with two candidates) using ACH than using unstructured reasoning. Not really overwhelming evidence either way, but – and this is hardly a surprise – it at least suggests that ACH doesn’t do any harm.

Some of the other data from his study might actually be more interesting – and useful – than the statistical benefits of ACH, or lack thereof. People who used it had a higher degree of confidence in their work, and exhibited dramatically (Brasfield says “tremendously”) fewer discernible signs of confirmation bias, one of the great-granddaddies of analytic failures. (Given that the subjects were apparently all intelligence students, this was possibly not a great cognitive bias to examine. Then again, given the large number of participants who exhibited signs of this…)

This suggests, to me, that ACH probably results in better-quality work products, even if the bottom line (i.e. accuracy) isn’t materially affected. So, yay.

Other tidbits from the thesis that I found interesting, though they have almost nothing to do with the main thrust of the research: As part of studying cognitive bias, all participants were queried for their political affiliation. Not too surprisingly, the majority of students identified as Republicans, though the numbers weren’t quite as lopsided as I’d expected. (There’s a kind of stereotype of liberals in the intelligence community as being… largely nonexistent.)

Much more surprisingly…

“Among Democrats, the percentage of participants who forecasted in favor of Gregoire(D) compared to Rossi(R) was strongly in favor of Gregoire and remained nearly identical from the control to experimental [i.e. ACH – Nemo] group. While this might suggest the effects of a [political] mindset were prevalent in both groups, it is more likely this appears to be the case not because of the influence of an actual mindset, but because Democrats overwhelmingly forecasted correctly in both groups.

(Emphasis mine, natch.) Well, Democrats do insist they’re the “reality-based” party, so…

Published in: Geekiness, General | on September 28th, 2010| 1 Comment »

Both comments and pings are currently closed.

One Comment

  1. On 10/11/2012 at 12:43 pm belong Said:


    ยป ACH, Cognitive Bias, and Politics – Entropic Memes…