Regression Fallacy

Category: Probability & Belief

The regression fallacy is when you credit (or blame) some intervention for a change that was going to happen anyway because of regression toward the mean. Extreme results tend to be followed by more average ones for purely statistical reasons, but you invent a cause: the treatment, the punishment, the pep talk, the new coach.

How it works

Most real-world outcomes are a mix of a stable component (skill, true value) and a random component (luck, noise, measurement error). When you observe an extreme value, it is extreme partly because the random part happened to line up in one direction, and that alignment almost never repeats. So the next measurement drifts back toward the true average. This is regression toward the mean, and it is a mathematical near-certainty, not a force. The fallacy is watching that drift and attaching a cause to it: you selected a group precisely because it was extreme, you did something, the group regressed, and you conclude the something worked.

Where you'll see it

  • Kahneman and Tversky's flight instructors (1973): Israeli Air Force instructors were convinced that harsh criticism improved the next landing and praise ruined it. Kahneman realized both were pure regression. He later called it one of the most important insights of his career, because it means the world punishes you for rewarding people and rewards you for punishing them, entirely by accident.
  • Horace Secrist's 'The Triumph of Mediocrity in Business' (1933): after ten years and 200-plus charts, Secrist 'proved' that top firms decline toward mediocrity and weak firms rise toward it, blaming competition. Harold Hotelling's review pointed out he had proved nothing but regression to the mean. Secrist had, in Hotelling's framing, proved the multiplication table by arranging elephants in rows.
  • The Sports Illustrated cover 'jinx': athletes on the cover after a career-best streak tend to slump afterward. Fans blame the curse of the cover. You get the cover for an extreme peak, and extreme peaks regress. Same logic behind the rookie-of-the-year sophomore slump.
  • Medical and policy evaluations: a clinic recruits patients with the highest blood pressure, gives a treatment, and sees pressure drop at follow-up. Some of that drop is regression, because you selected the most extreme readings. Without a control group you cannot tell how much of the improvement was the drug and how much was the mean pulling back.

Where it comes from

Francis Galton discovered the underlying phenomenon in 1886, measuring the heights of 928 adult children against their parents and finding that tall parents had children closer to average, and short parents likewise. He called it 'regression towards mediocrity in hereditary stature,' which is where the word 'regression' in statistics comes from. The leap from statistical phenomenon to named reasoning error came later. Harold Hotelling's 1933 demolition of Horace Secrist's business book is the canonical worked example, and Daniel Kahneman and Amos Tversky's 1973 paper 'On the Psychology of Prediction' established it as a systematic cognitive bias, showing people ignore regression when making intuitive predictions.

How to counter it

Imagine the no-treatment world. Before you credit the intervention, ask what would have happened if you had done nothing. If you only intervened because a value hit an extreme (worst quarter, sickest patient, hottest streak), the honest baseline is movement toward average, not staying put.

Demand a control group. The only way to separate your effect from regression is comparing against a group that was equally extreme but got no treatment. Any before-and-after study that selected on an extreme and skipped a control is contaminated by regression, full stop.

Watch for selection on the extreme. The fallacy needs one specific setup: you pick cases because they are unusually high or low, then measure again. Whenever you notice you started from a peak or a trough (top salespeople, failing schools, a record month), flag that any subsequent 'improvement' or 'decline' is suspect.

Separate skill from noise. Estimate how much of the outcome is stable versus luck. The noisier the metric, the harder it regresses, so a single quarterly number or one test score will regress far more than a ten-year average. Trust aggregates over dramatic single data points.

The tell

You catch yourself saying 'ever since we did X, things went back to normal' about a metric you only started tracking because it was at an extreme. The intervention always looks like it works, and it always kicks in right after the worst (or best) moment.

Related biases

References

  1. Francis Galton (1886). Regression towards Mediocrity in Hereditary Stature. Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246-263
  2. Harold Hotelling (1933). Review of The Triumph of Mediocrity in Business by Horace Secrist. Journal of the American Statistical Association, 28(184), 463-465
  3. Horace Secrist (1933). The Triumph of Mediocrity in Business. Bureau of Business Research, Northwestern University
  4. Daniel Kahneman & Amos Tversky (1973). On the Psychology of Prediction. Psychological Review, 80(4), 237-251
  5. Daniel Kahneman (2011). Thinking, Fast and Slow (Ch. 17, Regression to the Mean). Farrar, Straus and Giroux