Simpson’s Paradox and the Trickiness of Statistics

Data are everywhere. They are used to inform, they are used to make statistical arguments for one thing and against another, and much, much more. But they are tricky. Mark Twain is quoted as saying that “facts are stubborn, but statistics are more pliable.” That pliability allows them to be used in tricky ways, both intentionally and unintentionally. One tricky thing to keep an eye out for is Simpson’s Paradox.

Simpson’s paradox occurs when you look at groups of data and see one thing, but when you combine the groups you see something else. For a simple, common example one can look to sports. In baseball, you might have one batter – batter #1 – that every year has a higher average than batter #2. But when you combine their three years of batting, batter #2 has a higher batting average. The paradox happens because data are not weighted. Batter #2 had one great year (2011) which also happens to be the year that Batter #2 had the most at bats.

A slightly different example is with U.S. Department of Education National Assessment of Educational Progress (NAEP). Simply, when looking over the past 15 years of NAEP test data all combined together, scores look somewhat stagnant. There is no improvement. However, when you separate the scores by race and ethnicity, almost all groups are improving. How can that be? Because the weights are changing. Demographics are changing. There are more Hispanics/Latinos taking the NAEP test, and while their scores are increasing, their scores are still below the average. The effect is that it appears as though all students’ NAEP scores are remaining steady.

The lesson here is that when looking at data, be careful. And be particularly careful when looking at data that attempt to explain too much with too little.

• Categories:

  • Education