When comparing Kansas school test scores to those of other states, it’s important to consider disaggregated data. Otherwise we may — figuratively speaking — let the forest obscure the trees.
Kansas school leaders are proud of Kansas schools, partly because of scores on the National Assessment of Educational Progress (NAEP), known as “The Nation’s Report Card.” Kansas ranks pretty high among the states on this test. It’s important, however, to examine the results from a few different angles to make sure we understand the entire situation. An illustrative video is available here, or at the end of this article.
Data for the 2013 administration of the test was just released. I’ve gathered scores and made them available in a visualization that you can use at wichitaliberty.org. The most widely available NAEP data is for two subjects: reading and math, and for two grades, fourth and eighth. The video presents data for Kansas, Texas, and the average for national public schools. I choose to compare Kansas with Texas because for several reasons, Kansas has been comparing itself with Texas. So let’s look at these test scores and see if the reality matches what Kansas school leaders have said.
Looking at the data for all students, you can see why Kansas school leaders are proud: The line representing Kansas is almost always the highest.
NAEP makes data available by ethnic subtypes. If we present a chart showing black students only, something different appears. Now Texas is higher than Kansas in all cases in one, where there is a tie.
If we consider Hispanic students only: Texas is higher in some cases, and Kansas and Texas are virtually tied in two others. National public schools is higher than Kansas in some cases.
Considering white students only, Texas is higher than Kansas in three of four cases. In some cases the National public school average beats or ties Kansas.
So we have what seems to be four contradictory statements, but each is true.
- When considering all students: Kansas scores higher than Texas.
- Hispanic students only: Kansas is roughly equal to Texas.
- Black students only: Kansas scores below Texas.
- White students only: Kansas scores below Texas in most cases.
How can this be? The answer is Simpson’s Paradox. A Wall Street Journal article explains: “Put simply, Simpson’s Paradox reveals that aggregated data can appear to reverse important trends in the numbers being combined.”
The Wikipedia article explains: “A paradox in which a trend that appears in different groups of data disappears when these groups are combined, and the reverse trend appears for the aggregate data. … Many statisticians believe that the mainstream public should be informed of the counter-intuitive results in statistics such as Simpson’s paradox.”
In this case, the confounding factor (“lurking” variable) is that the two states differ greatly in the proportion of students in ethnic groups. For example, in Kansas, 69 percent of students are white. In Texas it’s 33 percent. This large difference in the composition of students is what makes it look like Kansas students perform better on the NAEP than Texas students.
But looking at the scores for ethnic subgroups, which state would you say has the most desirable set of NAEP scores? It’s important to know that aggregated data can mask or hide underlying trends.
Here’s a question for you: Have you heard Kansas school leaders talk about this?