Comparing CATS to Better Tests

The new CATS results positively beg comparison to the much better information we get from the EXPLORE and PLAN tests.

All Kentucky eighth and tenth grade students now take EXPLORE and PLAN, respectively. These better tests come from the highly respected organization that creates the ACT college entrance test and are closely aligned to that test. Subjects comparable to CATS include math, reading and science.

Like the ACT, EXPLORE and PLAN provide information about how many students reach “Benchmark” scores that indicate students are on track to earn passing grades in their first college courses. In addition, because the demands of better paying non-college jobs have increased, the ACT says that many employers for those better paying jobs now want exactly the same skills that are needed to succeed in college. So, the Benchmark scores are more like measures of what is needed for a decent adult life. Thus, it isn’t out of line to say that the Benchmarks can fairly be considered a real measure of proficiency. And, that opens the door to compare the percentages of students reaching EXPLORE and PLAN benchmarks to the percentages that CATS says are “Proficient” or more.

Here is a quick rundown of what we found when we did those comparisons

Middle Schools (2006-07 Percent / 2007-08 Percent)

Math
Meets EXPLORE Benchmark – 26% / 27%
Meets CATS Benchmark – 52% / 57%
Difference – 26% / 30%

Reading
Meets EXPLORE Benchmark – 35% / 33%
Meets CATS Benchmark – 66% / 67%
Difference – 31% / 34%

Science
Meets EXPLORE Benchmark – 11% / 10%
Meets CATS Benchmark – 56% / 60%
Difference – 45% / 50%

In both reading and science, the percentage meeting the EXPLORE benchmark went down this year while CATS proficiency rates increased. In middle school math, while the percent reaching the benchmark went up slightly, the rise in the CATS proficiency rate was much larger.

The differences in proficiency rates from 2006-07 to 2007-08 increased for all subjects, indicating that CATS scoring for middle schools got even easier this year.

High Schools (2006-07 Percent / 2007-08 Percent)

Math
Meets Explore Benchmark – 23% / 21%
Meets CATS Benchmark – 39% / 39%
Difference – 16% / 18%

Reading
Meets Explore Benchmark – 41% / 40%
Meets CATS Benchmark – 60% / 60%
Difference – 19% / 20%

Science
Meets Explore Benchmark – 15% / 15%
Meets CATS Benchmark – 42% / 41%
Difference – 27% / 26%

First of all, notice that the differences in claimed proficiency rates for all subjects are much lower for high schools, indicating the high school CATS academic tests (known as the Kentucky Core Content Tests), while still inflated, are much better aligned with real world needs than the middle school tests.

Also notice that except for science, there was a small creep in grade inflation even in the high school CATS scores as shown by the increase in the differences in proficiency rates from 2006-07 to 2007-08.

Overall, several things are clear. CATS scoring is generally getting inflated, something we have shown previously using other tests.

Furthermore, CATS scores of “Proficiency” clearly mean something less demanding in middle school than in high school. Thus, it isn’t surprising that high schools show the worst performance on CATS. This does not mean that middle schools are doing a better job. It just means middle school CATS is even more inflated than the high school CATS tests.