CATS Assessment and Accountability Task Force Meeting 3

First, my apologies for the length of this post. As this meeting got no coverage elsewhere, I believe our readers deserve to see this expanded discussion.

There wasn’t much consensus in the CATS Task Force, as it is commonly being called, during the group’s third meeting on September 18, 2008. In fact, the traditional news community voted on the likely importance of this group to Kentucky’s public education scene – by not even showing up – again.

Overall, agreement on writing is limited to a general acceptance that writing instruction is important and that writing portfolios are useful instructional tools. However, this group’s consensus quickly breaks down on the very important question of keeping writing portfolios in the state’s CATS school assessments. Opinions generally center in one of two very different camps.

One group believes that if the portfolio program is removed from CATS, then Kentucky’s teachers will stop teaching writing. This group firmly believes that only what gets assessed will be taught. The “Keep Them” group thinks the portfolio program mostly just suffers from administrative and management problems and only needs improvement in things like better instruction to teachers about how to manage this CATS element. Key spokespeople for this position include State Representative Harry Moberly and Marion County School Superintendent Roger Markum, who also represents the Kentucky Association of School Superintendents.

A second group believes that writing portfolios should be removed from CATS. The “Remove Them” group believes arcane rules intended to prevent teachers from giving students too much help on portfolios actually wind up inhibiting effective instruction. This group never the less believes that portfolios should remain in the instructional program. However, the group thinks teachers can be encouraged to do this by non-assessment programs such as audits. Key members of the “Remove Them” group include State Senator Dan Kelly and Kentucky Education Association President Sharron Oxendine.

Unfortunately, both groups argued more from beliefs and assertions than actual evidence during the meeting. Part of the reason was a very relevant report requested by Steve Stevens, head of the Northern Kentucky Chamber of Commerce, at the previous task force meeting wasn’t presented, or even mentioned. Stevens wanted research into the fact that Tennessee outscored Kentucky for eighth grade writing on the National Assessment of Educational Progress (NAEP) in 2007. The Bluegrass Institute has written about this subject before, and we see Kentucky’s rather poor performance on NAEP writing as evidence that teachers in other states are teaching writing – much more effectively – without needing portfolios in their state accountability systems.

Certainly, the task force’s portfolio work will be incomplete without considering the remarkable fact that Tennessee outscored Kentucky for writing proficiency although Tennessee has no portfolios in its accountability program. In fact, our further investigation of the 2007 NAEP Grade Eight Writing Assessment shows that Kentucky only did statistically significantly better than just five states among the 45 states that participated. Kentucky was statistically significantly outscored for writing proficiency by 22 states. This provides compelling evidence that it isn’t essential to have writing portfolios in state assessments in order to have good writing programs.

Some other information presented at the meeting also raised questions about Kentucky’s accountability program with portfolios. Ken Draut, and associate commissioner at the KDE, discussed an audit of the accuracy of the 2006-07 writing portfolio scoring. The audit shows an old problem continues – portfolio scoring continues to be inflated compared to the on-demand writing samples that are collected during the CATS academic tests.

Inflation is particularly evident for the higher CATS portfolio scores of “Proficient” and “Distinguished.” Data presented by Draut showed that out of 588 portfolios that were originally graded with the top score of “Distinguished,” the audit found only 37, just six percent, actually deserved that top grade. Incredibly, out of the same original “Distinguished” score group of 588, 176, or 30 percent, should have received the much lower score of “Apprentice,” according to the auditors.

There was another important message hidden in the CATS writing data that Draut presented, and it didn’t make it into discussion, either. Across all school levels, the overall writing academic indexes and the portfolio scores increased from last year. However, the on-demand writing proficiency rates in 2007-08 for both elementary and middle schools actually dropped from the previous year. On-demand writing is assessed under test conditions during the CATS academic test periods. The results are scored by independent graders, unlike portfolio scoring that is done by teachers.

By the way, the audit situation provides a clue to an unspoken reason why some may be fiercely protecting keeping writing portfolios in the accountability program. Because portfolio scores are inflated, if portfolios are removed from CATS, the overall school accountability index scores will decline.

There are definitely people on the committee whose primary focus seems to be not doing much of anything to CATS before 2014, be it with portfolios or anything else. Though he was absent from the meeting due to illness, Commissioner of Education Jon Draud previously stated he doesn’t want to make many changes. At this third meeting of the task force, State Board of Education Chair Joe Brothers also raised concerns that making any significant changes to CATS now would require a new three-year cycle to develop new scoring targets for schools. These hold-the-line advocates argue that changes will make it harder for schools to meet their CATS goals in 2014. Left unsaid, however, was the really critical question – are the current CATS goals truly valid targets worth preserving for another six years?

While portfolios took up the lions’ share of the time, there was some discussion on another contentious CATS issue – assessment of arts and humanities. It was pointed out that the CATS arts and humanities assessment may actually be working to diminish instruction in the performing arts. The test questions apparently have degenerated into little more than an arts vocabulary test, but actual demonstrations of students playing instruments or showing off sculpture and paintings are totally absent from CATS. There seemed to be fairly widespread interest in a pilot program to change the CATS arts assessment, but issues of funding could be a significant impediment to a statewide implementation. Interestingly, no-one seemed concerned that changing the arts assessment might require a new three-year adjustment time to develop new scoring targets. That argument only came up during the portfolio debate, making me wonder if this is more of a smoke screen than a legitimate concern.

At the close of the meeting, discussion facilitator Dave Spence was unable to get any consensus beyond a universal desire for more information. Hopefully, that will include gathering information about NAEP writing in addition to other areas of interest. And, hopefully, the next meeting will be more data, rather than opinion, driven.

Finally, there was a request from the committee for all interested parties to submit comments and ideas to the committee. If the KDE sets this up in an orderly way, I’ll pass details on how to submit comments in a later post. Until then, you can always send this to the public affairs office at the KDE, and we would certainly appreciate receiving a copy of your thoughts, as well. I hope the popular press finally wakes up and broadcasts this request, as well. It would be unfortunate if many valid concerns about CATS go unheard simply because the informed members of the public didn’t know their input was requested.