Prichard still doesn’t get it about state comparisons with federal tests

Prichard blithely ignores sampling errors in federal testing

I guess the Prichard Committee for Academic Excellence will never get it – you cannot do simplistic state to state rankings with the National Assessment of Educational Progress (NAEP). There is too much sampling error in those federal scores.

Prichard has been jumping on the mathematically challenged bandwagon of state rankings with NAEP for years. They just did it again following a brand new report from Education Week that claims Kentucky ranks “11th in percent proficient in grade 4 reading.”

Kentucky ranked 11th? Really?

Here is a map I assembled from the Main NAEP Data Explorer that compares Kentucky to other states for the percentage of fourth grade students who scored Proficient or Above on the 2009 NAEP reading assessment. This map DOES correct for sampling error in NAEP results.


What does this map tells us?

First, it is likely (statistical confidence level of 95 percent) that only a handful of states in the Northeast, shown in green, outscored Kentucky on this assessment. That sounds great.

But, in addition, we know that Kentucky got statistically insignificantly different scores (i.e. only tied) a whole bunch of other states shown in tan. My quick count shows there are about 26 of those.

Kentucky did outscore other states, shown in salmon color, around 19 or so.

So, how does Kentucky rank?

The only thing NAEP can really tell us is that Kentucky ranks somewhere in the very big middle.

And, it is a very big middle. If the NAEP were more accurate, we actually might rank as high as fifth in the nation. On the other hand, we might rank as low as 31st place, or so.

Boy, there’s real precision.

Keep in mind, grade four reading is our best subject and grade for NAEP reading and math results. It gets worse for mathematics. There are also some surprises when we look at results only for poor whites (click here to see maps that explore that in our freedomkentuky.com Wiki).

One thing we do know to an accuracy of a couple of percentage points: only 36 percent of Kentucky’s fourth grade readers tested Proficient or Above on the NAEP in 2009. That’s little more than one in three of our students. If that is an exemplary performance compared to other states, the whole country is in very serious trouble. Even being first in that league may be nothing to crow about.