What do NAEP scores for learning disabled students really show?

– Quick answer – no-one is really sure

Over at the Prichard Blog they are happy about a simplistic comparison of the performance of Kentucky’s students with learning disabilities on the federal government’s National Assessment of Educational Progress (NAEP) tests.

Let’s temper that enthusiasm a bit.

As recently reported in Education Week (subscription), the people running the NAEP are uncomfortable about the way they test these kids and how many of them get excluded from the tests. The governing board that oversees the NAEP has been uncomfortable for a decade, ever since I pointed out issues about uneven exclusion rates from state to state on the NAEP 1998 reading assessment.

Right now, no one knows how a number of factors might be impacting the validity of the scores for these special kids, not even the people running the NAEP program.

So, approach those NAEP learning disabled scores with caution – the people running the program do.



There are a number of issues that could impact the validity of State NAEP scores for students with learning disabilities. Here is a short discussion of a few of them.

First, the rules on who gets identified as learning disabled vary from state to state. Thus, for the recent 2009 NAEP fourth grade mathematics assessment, the percentage of each state’s raw sample of students that were identified as learning disabled varied from a low of 10 percent of all students in the state to a high of 20 percent. That is a very large difference, which raises questions about whether some states over/under identify true learning disabled students.

If kids who really are not learning disabled are unevenly included in the mix in some states, then the learning disabled scores might become inflated. That is especially true if those kids get special testing accommodations for learning disabled students that regular students are not allowed to have.

Secondly, the percent of each state’s raw sample that gets excluded because they are considered too disabled to take the NAEP also varies widely. In some states, only one percent of the entire sample got excluded in the 2009 NAEP fourth grade math assessment, while in others the rate was as high as five percent. Kentucky’s fourth grade exclusion on that assessment was 3 percent, above the national average of two percent.

If you exclude more of your weakest students, your scores are going to get inflated. The problem here is while we know the percentages of kids excluded varies, no-one knows for sure how much that impacts scores. Research to date in this area has not included scientific split sample models and is therefore unconvincing and continues controversial.

A third problem involves the testing accommodations used in the NAEP.

Those accommodations can include such things as having people read the tests to the student (not allowed for NAEP reading, but legal on all of Kentucky’s state assessments and on other NAEP assessments), providing extra time, and even providing someone to write down the student’s answers. They can contribute to inflated scores if improperly used. They are also easy to abuse (scribe to student – “Are you SURE that is your final answer?”).

There has been little, if any, real scientific research into whether or not accommodations might be inflating scores. The decision to go completely to accommodated testing in NAEP was a policy decision made without adequate research. That is why, years later, the governing board for the assessment is still troubled about the situation, as Education Week reported.

Accommodations could give Kentucky another unfair advantage. Kentucky’s state assessments have a much longer history of using accommodations in testing than most other states. So our teachers and students have a lot of experience with accommodations that could create an advantage that isn’t present elsewhere. That could corrupt the meaning of our scores, as well. Again, I am not aware of any research on this familiarity issue.

Finally, NAEP is a sampled test, pulling somewhere around 3,000 students in Kentucky for each assessment at each grade level. Because learning disabled students form only a portion of the sample, the relatively small size of the group leads to rather notable statistical sampling errors. What looks like a big score difference might not be.

Furthermore, Kentucky’s very non-standard student demographics – our students run about 85 percent white versus a much lower national percentage – sometimes gives Kentucky unfair advantages in NAEP comparisons.

The data Prichard used includes both true students with learning disabilities and those students who are in a closely related, but different category known as the federal 504 program. I think the numbers Prichard cites also include English Language Learners (ELL), which again gives Kentucky an advantage because we have few ELL students here and they tend to score much higher than ELL in states with more heavy immigration.

I assembled the set of tables below with the NAEP Data Explorer.

Table 1 below separates the scores for students with learning disabilities only (SD), as shown in the orange frame, from the ELL students.


Note that Kentucky’s true learning disabled students (without the 504 plan students included) only scored four points above the national average for similar public school students. Also note that Kentucky has hardly any learning disabled students who are also ELL. In fact, there were so few of them in Kentucky that the NAEP won’t even report their scores.

However, across the rest of the nation, the learning disabled ELL score notably lower than other learning disabled students (3 points lower). That lack of very low-scoring ELL in Kentucky gives the Bluegrass State an unfair advantage in the comparison of the numbers Prichard cites.

By the way, as the following figure shows (also assembled from the NAEP Data Explorer), the four point score difference for Kentucky versus the national SD students isn’t statistically significant.


Here is a table with the numbers Prichard cited.


Notice the numbers Prichard cited include the 504 plan students, who are not really learning disabled students.

Clearly, the real situation here is quite complex and not nearly the same as Prichard’s simple analysis might lead you to believe.

Even the people running the NAEP have doubts.

That is why they have been running research to try and figure out what is really happening for a decade – and these NAEP experts still are not set on answers.