Kansas school test scores, two interactive visualizations

When comparing Kansas school test scores to those of other states or the nation, it’s important to consider disaggregated data. Otherwise, we may make inaccurate conclusions regarding Kansas schools.

Kansas school leaders are proud of Kansas schools, partly because of scores on the National Assessment of Educational Progress (NAEP), known as “The Nation’s Report Card.” Kansas ranks pretty high among the states on this test. It’s important, however, to examine the results from a few different angles to make sure we understand the entire situation.

I’ve gathered scores from the 2013 administration of the test, which is the most recent data available. I present them in a visualization that you can use yourself through the links at the end of this article. The most widely available NAEP data is for two subjects: reading and math, and for two grades, fourth and eighth. In the nearby images captured from the visualizations, I present data for Kansas and the average for national public schools. The numbers are the percent of students that are at or above proficient.

Looking at the data for all students, you can see why Kansas school leaders are proud: The line representing Kansas is almost always the highest. But there are important statistical considerations to take into account.

NAEP makes data available by ethnic subtypes. If we present a chart showing black students only, something different appears. In some instances the line for national public schools coincides with the Kansas line, or is above the Kansas line. A similar pattern exists when considering Hispanic students only.

Perhaps surprisingly, when considering white students only, the same pattern exists: In many cases national public schools white students score as well as, or sometimes above, Kansas white students.

Looking at the data subgroups by eligibility for free or reduced price lunches is useful, too. This eligibility is a commonly-used surrogate for selecting students from low-income households. When looking at the subgroups, the advantage of Kansas schools sometimes disappears, although the effect is not as marked.

How can this be? The answer is Simpson’s Paradox. A Wall Street Journal article explains: “Put simply, Simpson’s Paradox reveals that aggregated data can appear to reverse important trends in the numbers being combined.”

The Wikipedia article explains: “A paradox in which a trend that appears in different groups of data disappears when these groups are combined, and the reverse trend appears for the aggregate data. … Many statisticians believe that the mainstream public should be informed of the counter-intuitive results in statistics such as Simpson’s paradox.”

A more technical paper gives this definition: “Simpson’s paradox refers to a phenomena whereby the association between a pair of variables (X, Y ) reverses sign upon conditioning of a third variable, Z, regardless of the value taken by Z.”

In this case, the confounding factor (“lurking” variable) is that Kansas differs greatly from national public schools in the proportion of students in ethnic groups. Most prominently, in Kansas, 68 percent of students are white. For national public schools, the value is 51 percent.

This large difference in the composition of students is what makes it look like Kansas students perform better on the NAEP than the national average. But looking at the scores for ethnic subgroups, can we still say that Kansas schools outperform national public schools? It’s important to know that aggregated data can mask or hide underlying trends.

Note that there is not much difference in eligibility for free or reduced lunches between Kansas and national public schools. This is why Simpson’s Paradox is not strongly apparent in these scores.

Have you heard Kansas school leaders talk about this? Or do they present Kansas NAEP test scores without considering the different makeup of the states?

The interactive visualizations of NAEP scores are not difficult to use. The adjustment most people may want to make is selecting a different combination of states. Click here to access the visulizations.