Skip to content

Data Wrangling·Lesson 34 of 67·intermediate

Exploratory analysis without fooling yourself

Simpson's paradox, survivorship bias, small-group noise and the multiple-comparisons trap — the four ways a correct calculation tells you something false.


In this lesson · 20 min

  • →Look at distributions and group sizes before believing an average
  • →Recognise Simpson's paradox and check for a confounder
  • →Spot survivorship bias in a filtered dataset
  • →Quantify how much a small group's number can be trusted
  • →Avoid finding a false result by slicing until something looks significant

Every calculation in this module is easy to get arithmetically right and still be misled by. This lesson is about the four failures that produce a correct number and a wrong conclusion.

Start with the shape, not the mean

PythonPython not started

Both average about 0.70. One model is mediocre at everything; the other is excellent at 70% of questions and completely fails the rest. They need different work, and the mean cannot tell them apart.

PythonPython not started

Simpson’s paradox

A model can be worse overall and better on every single subgroup:

PythonPython not started

Model B wins on easy questions (0.95 against 0.90) and on hard questions (0.60 against 0.50), and loses overall. Nothing is wrong with the arithmetic — the two models were simply given different mixes of questions:

PythonPython not started

Survivorship bias

The rows that are missing are often the rows that matter:

PythonPython not started

Reporting 0.72 over the answered subset is not a lie, but it answers a question nobody asked. The number a user experiences counts the failures.

Small groups are noise

PythonPython not started

Sinhala tops the table at 100% — from three questions, with an interval running from 0.44 to 1.00. It is entirely consistent with that model being worse than every other language.

Slicing until something looks real

If you test twenty slices at the 5% level, you expect one false positive even when nothing is happening:

PythonPython not started

An 18-point gap, entirely from chance. If you had gone looking for the slice where the model underperforms, you would have found one and written a ticket about it.

Correlation is not the mechanism

PythonPython not started

Slow answers score badly — but making the model faster would not help. Difficulty drives both.

A checklist

PythonPython not started

Try it yourself

Exercise 1

Compute overall (accuracy per model over all questions) and by_difficulty (accuracy per model and difficulty), then set paradox to True if one model wins overall while the other wins every subgroup.

Your turnPython not started
Exercise 2

Write trustworthy(correct, total, width=0.2): return True only when the Wilson interval around the accuracy is narrower than width.

Your turnPython not started

Check yourself

Model B beats Model A on easy questions and on hard questions, but loses on the pooled number. What should you check first?

Check yourself

You check twenty subgroups for underperformance and find one that is clearly worse. What is the responsible next step?

Recap

  • Look at the distribution first. Two models with the same mean can need completely different work.
  • Simpson’s paradox is the normal case in model comparison: check the group sizes match, and reweight or compare within subgroups when they do not.
  • Ask what got filtered out before the number was computed — the unanswered questions are usually the interesting ones.
  • Report group sizes or intervals. A 100% from three examples is consistent with being the worst group you have.
  • Choose your slices before you look, and confirm anything you find by exploring on fresh data.
  • A correlation usually has a third variable behind it; check within a narrow band of the suspected confounder.