Governor cites Prichard nonsense

Governor Steve Beshear treated the Boone County Education Foundation to nonsense education statistics a few days ago.

As reported by the Kentucky Enquirer in “Beshear touts education,” the governor touted Kentucky’s supposed dramatic improvement in the federal National Assessment of Educational Progress (NAEP) testing program compared to other states.

Well, as we’ve pointed out before, those rankings of NAEP performance, which the governor got from the Prichard Committee for Academic Excellence – not a proper state agency – are mathematical nonsense.

Kentuckians deserve better data and a more reasoned evaluation of the performance of our education system.



First of all, the newspaper article refers to NAEP reading scores from 2008. I can’t tell for certain if that was the governor’s error, the Prichard Committee’s error, or a reporting problem, but the NAEP didn't do any state reading testing in 2008.

The scores probably relate to 2009 NAEP results, so I’ll discuss some of the results for that year later. Rest assured, there is much more than a simple date error here.

The ranking is statistical nonsense

The NAEP is a sampled test. Only a small proportion of our students take it. As a consequence, the NAEP scores have enough plus and minus statistical sampling error to make Prichard's simplistic rankings invalid.

The truth is, once you consider the plus and minus errors present in each state's NAEP scores, it is difficult to determine exactly how each state really ranks. At best, you can only get a very fuzzy idea about how states compare to each other.

For example, I did a little work using the NAEP Data Explorer tool (a very useful, though complex, tool which allows checking for statistical significance in scores). My research shows that in 2009 grade 4 reading, Kentucky's whites might rank anywhere from 40th place to 11th place compared to whites in other states plus Washington, DC (see map below). The statistical sampling error in the NAEP makes it impossible to tie this down more closely. How's that for a lack of precision?

It’s the demographics

Why did I talk about white scores, and not overall scores such as the governor used?

An additional problem with the NAEP, which Prichard likes to conveniently ignore, is that when you look at the results, you simply must look at the scores disaggregated by race.

That’s not just my opinion. The people who run the NAEP now tell us this.

The new NAEP Science Report Card specifically points this out – using Kentucky as a specific example – with this graph on page 32.
Notice that while Kentucky’s overall score for eighth-grade science is statistically significantly higher than the nation (shown by the asterisk), the score for our white students is statistically significantly LOWER than the national white average.

How can that be, when whites comprise about 85 percent of our student population?

The answer is that other races in other states form a much larger share of the school populations there, and those other races score much lower than whites. So, if you only do simplistic comparisons of overall average scores, the demographics in other states put them at a disadvantage to a state like Kentucky where there are relatively few minorities.

The bottom line on demographics – you cannot simplistically compare Kentucky's NAEP scores, where 85 percent of the students are white, to a place like California, where whites are now a minority and lower scoring Hispanics are the majority population. This is particularly true when many of those California Hispanics are still learning English.

It’s not that we are really doing much better. It’s that other states face growing educational challenges we don’t.

The people who run the NAEP know this. That is why Figure 32 and the related discussion were included in the 2009 NAEP Science Report Card.

I guess Prichard doesn’t read that sort of ‘stuff.’ Ditto for the governor.

If you exclude more learning disabled kids, of course you’ll have higher scores

There is still more. Kentucky excluded a higher percentage of learning disabled students than the national average in the 2009 reading assessments. That inflates our reading scores, which further clouds state comparisons.

To learn more

You can find a great discussion of all these NAEP issues along with some neat maps that show how the state really compares here.

If you are friends with the governor, you might suggest that he should take a look – and drop that Prichard nonsense. Kentuckians deserve more accuracy from their governor than this baloney.