Congruence Bias

Category: Probability & Belief

The tendency to test your favorite hypothesis directly (does the evidence I expect show up?) while skipping the tests that would rule it out or point to a rival explanation. You run experiments designed to say yes, not experiments designed to decide.

How it works

You start with a hunch that feels most likely, the "congruent" hypothesis, and you plan tests that would confirm it if true. The catch is that a positive result from a confirming test often fits several explanations at once, so it barely moves the needle. What separates hypotheses is the discriminating test, the one whose outcome would differ depending on which explanation is correct, and that is exactly the test congruence bias skips. Baron and colleagues found people specifically overvalue questions with a high probability of a "yes" under the leading hypothesis, even when a different question would be far more informative. It is confirmation bias narrowed to the moment of experiment design: not how you read evidence, but which evidence you bother to gather.

Where you'll see it

  • A doctor suspects strep, orders a strep test, it comes back positive, and treatment starts. Nobody checks that the positive result does not rule out a second infection producing the same fever, so the other cause gets missed.
  • Your web page is slow. You assume it is the database, add a cache, and it feels faster for an afternoon. You never profiled the actual request, so you never saw that a blocking third-party script was the real bottleneck all along.
  • A recruiter believes a candidate is great, so the interview becomes a tour of the candidate's strengths. No question is designed to expose a weakness, so a positive impression that any decent candidate would produce gets read as strong evidence.
  • In Wason's 2-4-6 task, you are told 2-4-6 fits a hidden rule. You guess 'even numbers ascending' and test 8-10-12, then 20-40-60, both get a yes, and you announce your rule with confidence. You never tried 1-2-3, which would have told you the rule was just 'any increasing sequence.'

Where it comes from

The intellectual root is Peter Wason's 1960 "2-4-6" task, where people given a hidden number rule tested it almost entirely by generating triples they expected to fit, rather than triples that could break it, and locked onto wrong rules as a result. The specific term "congruence bias" was coined by Jonathan Baron, Jane Beattie, and John Hershey at the University of Pennsylvania in their 1988 paper "Heuristics and Biases in Diagnostic Reasoning II" in Organizational Behavior and Human Decision Processes. Across six experiments they showed subjects overvalued questions likely to return "yes" under the most probable hypothesis, and that the bias shrank when alternative hypotheses or the probabilities of negative answers were spelled out for them. Baron framed it as a distinct heuristic driving diagnostic reasoning, not just a rewording of confirmation bias, though the two are close cousins.

How to counter it

Write down the rival before you test. Force yourself to name at least one competing hypothesis in a sentence before designing any test. Baron's own experiments showed the bias shrinks sharply the moment alternatives are made explicit, so the fix is literally writing "or it could be X" on the page.

Demand a diagnostic test, not a confirming one. For any test you plan, predict the outcome under your hypothesis AND under the rival. If both predict the same result, the test is worthless for deciding between them. Keep only tests where the two predictions diverge.

Hunt for the "no." Deliberately design at least one test that would return a negative result if your favorite is true. A hypothesis that has survived a genuine attempt to break it is worth something. One that has only ever been fed softballs is not.

Separate "it worked" from "it was the cause." When a fix appears to solve the problem, ask whether anything else changed at the same time and whether the problem would have resolved on its own. A single positive outcome rarely isolates one cause.

The tell

You catch yourself designing a test where you already know it will come out the way you want, and you feel a small flush of satisfaction planning it rather than dread. That comfort is the tell: a real diagnostic test should feel like it could embarrass you.

Related biases

References

  1. Baron, J., Beattie, J., & Hershey, J. C. (1988). Heuristics and biases in diagnostic reasoning: II. Congruence, information, and certainty. Organizational Behavior and Human Decision Processes, 42(1), 88-110
  2. Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology, 12(3), 129-140
  3. Beattie, J., & Baron, J. (1988). Confirmation and matching biases in hypothesis testing. Quarterly Journal of Experimental Psychology A, 40(2), 269-298
  4. Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175-220