Board of Education adopts fuzzier weighting for new state assessment program

Could adversely impact validity and reliability significantly

The Kentucky Department of Education just announced action by the Kentucky Board of Education to change important weighting of various assessment elements that will be used in the states new public school assessment and accountability system. This new assessment is scheduled to replace the state’s defunct CATS assessment, which developed credibility issues and was disbanded in 2009.

Unfortunately, in the process of suddenly shifting weightings around, the board has now created a potential monster, a program with a high reliance on assessment elements that have a high degree of subjectivity.

Even worse, the assessment elements getting added weight are going to be self-graded by the schools and only inadequately audited by the Kentucky Department of Education, opening the door wide to exactly the issues of score inflation that always plagued the schools self-grading their own students’ writing portfolios under the old CATS system.

It’s going to be a classic ‘fox guarding the henhouse’ situation, perhaps much worse than we had with CATS Writing Portfolios, if the board doesn’t step back and take a look at the big picture surrounding the assignment of weights in the new assessment program.



The table below outlines the three major elements, or ‘pillars,’ under consideration for the new assessment program. These include “Next Generation Learners,” “Next Generation Instructional Program and Support,” and “Next Generation Professionals.” Shown with each of the pillars are details of items that will be included as of the board’s actions on June 8, 2011 (A fourth pillar under consideration, Next Generation Schools/Districts” was not discussed or included in the weighting schemes at this time).


This next table shows a breakdown of the Next-Generation Learners pillar, which currently is by far the most fully defined and well-developed area. It is also clearly the most objectively scored, as well.


Now, here is the problem.

The weighting changes adopted by the board yesterday will lower the validity and reliability of the new assessment program because weight has been removed from the most objective area, “Next Generation Learners.” Instead, that weight is now shifted to the most subjective area, the “Next-Generation Instructional Programs and Support” pillar.

Furthermore, until yesterday the “Next-Generation Instructional Programs and Support” pillar was to be based on inevitably subjective reviews of three programs in each school covering Writing, Practical Living and Vocational Studies and the Arts and Humanities. Those three reviews are required by Senate Bill 1 from the 2009 Regular Legislative Session. All three areas were part of the accountability program in the old CATS assessment but proved incompatible with test-based evaluation techniques, which led to many unintended consequences that adversely impacted program quality.

Further blurring the program review area, the board added two more program reviews to the mix yesterday, one for World Languages and the second to examine the state’s generally defunct Primary Program. Adding even more subjective program reviews on top of what Senate Bill 1 requires is clearly problematic. However, these new additions led the board to the decision to shift weighting around.

Blurring this picture even more is the fact that neither World Language nor the Primary Program look like good candidates for program review approaches.

First, World Languages probably don’t even belong in the program review area. Languages are taught as academic subjects and are clearly most suitable for incorporation as objectively tested items in the Next-Gen Learners area, instead.

Second, the primary program is a wistful idea that just never worked out in the real world. Furthermore, a law passed about 10 years ago makes Primary essentially optional. Thus there is no way to evaluate performance of schools with the concept in a legal manner. If a school can show why it isn’t using primary, it cannot be penalized for that decision.

Primary really doesn’t work any better than traditional approaches. I looked at the 2008 performance of schools based on the degree to which they still use primary concepts. My conclusion: Based on the 2008 data, none of the various the multi-age environments currently used in Kentucky produce a notable advantage in academic performance over the traditional graded structure. In fact, most of the implementations result in slightly lower performance. Therefore, the current rather lax enforcement of the requirement to implement multi-age environments in Kentucky is probably wise.

Given this real world data, allowing the Kentucky Board of Education to again try to push Primary down our teachers’ throats would be to abandon the best interests of our children. This very misguided board action, which was popped on board members with little notice, needs to be retracted, ASAP.

One last note, in voting to increase weight on the program reviews yesterday, the board ignored the advice of our national testing expert committee, the NTAPAA. A staff note provided to the board members pointed out that the NTAPAA had concerns about the self reporting aspect of the program reviews when it comes to accountability.

The same group of experts noted that it would take very carefully designed rubrics to make the program reviews work and that the reviews will take significant time and work in data collection.

Unanswered questions: Do our schools have the capability to do this extensive work? Will they do it without succumbing to the temptation to inflate the findings? Will the limited audit plan announced by the Kentucky Department of Education prove sufficient? Will adding two more program reviews overload local school staff?