Kentucky’s 2009 NAEP reading performance – is it credible?

I raised some questions yesterday about Kentucky’s fourth grade reading performance in the newly released 2009 National Assessment of Educational Progress (NAEP) Reading Report Card.

Today, I’m going to point out some more issues, this time focused on the state’s eighth grade results.

This graph shows results from the two most recent NAEP eighth grade reading assessments and the corresponding test results from Kentucky’s 100 percent testing of all eighth grade students with the EXPLORE Reading Assessment from the ACT, Incorporated.


Notice in the graph that Kentucky showed virtually no change in the percentage of students reaching or exceeding the EXPLORE Benchmark score between the 2006-07 and 2008-09 school terms.

In sharp contrast, Kentucky’s reading performance on the NAEP experienced a very notable five point rise in the same time interval.

Very simply, these notably different results raise questions about the 2009 NAEP scoring.

It should be pointed out that the EXPLORE test frameworks and scoring did not change, to my knowledge, during the time interval shown in the graph.

In sharp contrast, the NAEP reading assessment was changed in 2009 as explained by this quote from the 2009 NAEP Reading Report Card:

“The Reading Framework for the 2009 National Assessment of Educational Progress replaces the framework first used for the 1992 reading assessment and then for subsequent reading assessments through 2007. Compared to the previous framework, the 2009 reading framework includes more emphasis on literary and informational texts, a redefinition of reading cognitive processes, a new systematic assessment of vocabulary knowledge, and the addition of poetry to grade 4” (Page 4).

While the 2009 NAEP Reading Report Card goes on to claim that the 2009 results can be compared to prior year’s results, the Kentucky EXPLORE and NAEP comparison shown in the graph above raises questions about that assertion.

Clearly, the disparity between Kentucky’s NAEP and EXPLORE reading performance invites further investigation of the 2009 NAEP scoring. The situation also raises concerns about the validity of the 2009 rise in Kentucky’s NAEP scores in grade eight.

Furthermore, if the NAEP frameworks were changed in ways that impacted the trend lines for eighth grade, a similar problem could exist in grade four, as well. Unfortunately, ACT does not offer a companion EPAS test for that level.



The data shown in the graph for each test year is for the same cohort of students. Kentucky conducts EXPLORE testing in the fall of each academic year and the NAEP conducts its testing a few months later in late winter. Thus, it should be understood that the data in the graph above for 2006-07 and for 2008-09 are both for the same cohort of eighth grade students. NAEP didn't conduct testing in the intervening school term.

It should also be emphasized that both the NAEP and the EXPLORE tests are controlled and scored by organizations outside of Kentucky. However, Kentucky does have some influence on the results from both tests because it has control on the use of testing accommodations. This results in some skewing of the results for both assessments.

For example, while few Kentucky students are excluded from EXPLORE, an appreciable proportion of the state’s students with learning disabilities receive test accommodations on EXPLORE that the ACT Incorporated does not allow on any ACT tests taken for college application purposes.

Kentucky’s accommodations program also impacts NAEP results, but in a somewhat different way. An appreciable proportion of Kentucky’s learning disabled students (48 percent in 2009 in grade four and 55 percent in grade 8) are completely excluded from the NAEP due to state requirements for incompatible testing accommodations that the NAEP will not allow. In fact, Kentucky’s learning disabled exclusion rate for NAEP reading in both fourth and eighth grade is among the very highest in the nation. Only two states had higher exclusion proportions in the 2009 eighth grade NAEP reading assessment.

This exclusion issue is largely due to a very controversial accommodation allowed in Kentucky – many of the state’s learning disabled students actually have all “reading” tests read to them. This totally changes the construct of the test from printed text decoding and comprehension to one of spoken word comprehension. Neither the ACT nor the NAEP will allow this clearly inappropriate accommodation on tests taken for record although EXPLORE does report scores for students tested this way for Kentucky’s internal state uses only.

Find more information:

About EXPLORE:

While many people are somewhat familiar with the NAEP, fewer probably know about the EXPLORE. Briefly, EXPLORE was developed by the ACT as a part of a coordinated college preparation testing program know as the EPAS system.

Kentucky tests all its students with EXPLORE in eight grade, the PLAN in tenth grade and then conducts 100 percent testing of all public school eleventh grade students with the ACT college entrance test.

Each of the EPAS tests has a set of “Benchmark” scores for the subjects of English, math, reading and science which are linked to an empirical study of how well ACT college entrance test scores compare to actual success in the first related college course. Students who score at or above the Benchmarks have a 75 percent chance of earning a “C” and a 50 percent chance they will get a “B” in those freshman college courses.

Learn more about the Benchmark Scores here.

ACT discusses the EPAS system here.

Information about the NAEP can be found in the 2009 Reading Report Card and in the NAEP Web Site.

Data sources used to assemble the graph:

2009 NAEP Reading Report Card, Table A-19

EXPLORE Excel Spreadsheet, “EXPLORE_Benchmark_06070910.xls” from Kentucky Department of Education.