CATS Writing Inflation

Over a week ago we said something about the dubious jump in high school on-demand CATS writing scores for 2008.

Now, another education-focused blog is picking up this issue, speculating about the reasons behind the implausibly big rise in high school writing scores from the on-demand tests that are given during CATS testing.

That blog makes some good points, but it misses an even more obvious issue that has nothing to do with changes in scoring for “Novice” and “Apprentice” grades in CATS. The more obvious issue is the huge increase in the number of students getting the top CATS scores of “Proficient” and “Distinguished.” Supposedly, there were no changes to the grading for these top scores, and the impacts from the obvious inflation are much more severe for elementary and middle schools than for the high schools.

Unlike the high schools, the elementary and middle school scoring for on-demand writing changed in 2007, not 2008. Also, the grades tested shifted from fourth to fifth and from seventh to eighth (Let no one say that the CATS is a stable testing program).

In any event, as the figures below show, the jump in the percentages of students graded “Proficient” or “Distinguished” between 2006 and 2007 in both elementary and middle schools is so extraordinary as to be completely unreasonable.

Statewide, Kentucky elementary schools suddenly exploded from having only 4.48 students graded “Proficient” and just 0.83 percent “Distinguished” in 2006 to 43.59 percent “Proficient” and 9.78 percent “Distinguished” in just one year. As you can see from the trends since 2001, this is an unprecedented and totally unreasonable change.



In middle schools, the trends are almost as severely unreasonable. Proficiency spiked from only 13.59 percent to 35.60 percent and the proportion graded “Distinguished” exploded by more than 600 percent (1.03 to 6.33 percent). Nothing in the middle school trends from 2001 to 2006 would predict such a massive, one-year change.



By the way, the National Assessment of Educational Progress (NAEP) tested eighth grade writing in 2002 and 2007. Kentucky’s change in writing proficiency rates on the federal test was statistically flat. The NAEP results indicate that the CATS writing was graded too hard in 2002 and then became grossly over-compensated by grading far too easy in 2007.

As I said earlier, let no one say that the CATS is a stable testing program, or credible, either.

How long will Kentuckians be inundated with such bloated nonsense while our education system struggles to make itself look good instead of moving out to do something that really makes a difference for our kids? Right now, entrenched education interests are defending the writing portfolio program to the hilt while evidence piles up from failures of similar programs elsewhere (like Vermont), and from the NAEP, and from several analyses of teacher opinions that overwhelmingly indicate the portfolios are part of the writing problem here.

And, inflating scores to hide the issue isn't going to make one kid in this state a better writer.