Fooling with figures

The choirmaster of the KERA Amen Chorus is at it again.

The Prichard Committee for Academic Excellence just released its “Kentucky Schools: Achieving the Top 20 by 2020: An Update.”

This little piece is a statistical cornucopia of apples and oranges comparisons.

Prichard’s paper starts out by making old news sound like new. It talks about “new results,” but some of the National Assessment of Educational Progress (NAEP) data in the paper date as far back as 2002 (NAEP Grade 4 writing). The NAEP science data is all five years old, as well. Who knows how Kentucky ranks on these today?

In another case, the data shown is identified as coming from the 2002 NAEP grade 8 writing assessment. It really comes from the 2007 assessment, but that is now three years old information. It isn’t new, either.

I have a lot of concerns about the way Prichard blithely ranks NAEP data. It’s not worth spending a lot of time on it, but I will address one representative example: Prichard’s claim that Kentucky ranks 9th for fourth-grade reading on the National Assessment of Educational Progress in 2009.

Are we REALLY that good? Can you tell it from the NAEP?

For starters, check out this graphic.


When we are talking about state to state performance, it is important to look at Kentucky’s white data because whites comprise about 85 percent of the total enrollment in our schools. This map shows states where white fourth grade student NAEP reading scores are statistically no different from Kentucky (highlighted in tan) along with the states that did better (highlighted in green) and worse (shown with salmon highlighting).

First, notice that NAEP cannot detect any differences in performance for white students between Kentucky and the 29 other states shown in tan. That’s because statistical sampling error in the NAEP makes it a rather blunt instrument to create state rankings (Prichard doesn’t talk about that).

Next, note that we really can only know with reasonable certainty that 11 states did worse than Kentucky, and we only know with certainty that 10 states did better. The only other thing we know about the huge majority of all Kentucky students is their performance fits – somewhere – in the enormous middle. That is all we can safely conclude from the 2009 NAEP grade four reading results for white fourth graders.

This example shows one reason why trying to rank the states on a range of 1 to 50 with the NAEP is rather silly. The NAEP doesn’t measure things closely enough to make that effort worthwhile, and that is why the NAEP reports themselves don’t provide such rankings. The experts who run the NAEP know that wouldn’t be a valid use of the results.

If you want to see more, such as how blacks in Kentucky did, and why you have to look at separated scores for whites when evaluating Kentucky’s NAEP performance, read on.



Here is a similar graphic for black only NAEP fourth grade reading performance in 2009.


Note that some states in the northwest, highlighted in slate gray, have so few black students that the NAEP would not publish scores because the statistical errors got too large.

In general, even where blacks were present, the statistical errors are pretty big, so Kentucky winds up tying almost all of those states. Only three states and the District of Columbia did worse than Kentucky for black fourth grade reading. On the other hand, the NAEP’s sampling errors are so large that we can only say with certainty that a handful of states did better.

Basically, once you consider the statistical errors, you can’t tell much of anything about state to state rankings for blacks. That doesn’t mean there are no notable differences. It just means that the NAEP isn’t sensitive enough to show them with a high level of confidence.

By the way, here is the map comparing the ranking for all students across the various states to Kentucky.


This looks more impressive for Kentucky. Many states did worse when all the student scores get averaged together.

But, how can that be when our whites and blacks didn’t do anything special compared to their peers?

The answer involves another problem with trying to rank states using the NAEP. Most states have a much lower percentage of whites than Kentucky does – MUCH LOWER. They also have much higher percentages of minority populations and English Language Learners (ELL). Those non-white and ELL students drag down the averages in the other states. Meanwhile, with its roughly 85 percent white, almost totally English as a first language population, Kentucky enjoys an incredible advantage in any simplistic comparison of overall NAEP scores. The NAEP offers no corrections for this, hence the graphic above makes Kentucky look really good.

However, that image starts to shatter once we break down the scores by race to compare apples to apples.

If you want to learn more about the pitfalls in trying to work with the NAEP, check out our freedomkentucky.org article about the National Assessment of Educational Progress.

And, watch out for those who try to fool you with figures.

Technical Note: The graphics above were assembled with the Main NAEP Data Explorer.