Wednesday, October 1, 2008

CATS Assessment and Accountability Task Force Meeting 4

Once again, there doesn’t appear to be coverage anywhere else on this ongoing Kentucky Department of Education effort. I didn’t see any reporters at the meeting.

The session started with a quick welcome from education commissioner Jon Draud, who called in by telephone from home as he is still recuperating from a recent stroke (Jon, all at the Bluegrass Institute wish you a speedy recovery).

The initial report at this meeting concerned the committee’s request for input from educators and the public that was issued after the last meeting. Lisa Gross indicated there were about 200 e-mail responses plus more in faxed returns. Just assembling the e-mails into one Word document required 100 pages.

Gross asked the committee if it would like a printout of this file or would prefer to get an electronic version. The electronic version was selected.

Unfortunately, when I asked to be included on this mailing, I was informed that the document was only for the committee, and it was up to the committee to release it to the public. That makes it tough for people who sent comments to see if what they wrote is accurately reflected in the edited document the committee is going to receive. This also precludes any public discussion about the comments. It’s hard to understand why this document, created for public discussion by a public committee, would be withheld. Perhaps this decision will be revisited.

The next issue was whether the committee wanted to hear live testimony from people who wanted to address the task force. It was pointed out that the task force has only a limited amount of time left and cannot afford to be bogged down in unsolicited presentations (Also, the public comment request provided an opportunity for individuals to transmit their thoughts to the committee). In the end, it was decided that the committee will continue to manage presentations to cover topics of interest only.

Next came a lengthy discussion about the assessment of arts and humanities in CATS. This continued deliberations from meeting three, which were covered in our notes on that meeting.

The task force appears to be generally positive about a pilot arts and humanities assessment program that has been tried in some elementary schools. However, a request at the last meeting for cost estimates to implement the program statewide at all school levels went unanswered. So, in the end, the committee deferred a vote on a draft recommendation to change the arts and humanities assessment pending that cost analysis, which is now promised for meeting five.

One positive note came in comments from Dave Spence from the Southern Regional Education Board (SREB), who is the facilitator at the task force. He said that SREB received a grant from the Gates Foundation to look at college readiness standards in the P to 12 systems in Kentucky, Texas and Virginia. The research will look at state standards in language arts and math and how the state tests that. It will also address accountability and curriculum changes and the teacher professional development required to make this all work so our kids will be better prepared for college.

The rest of the meeting was completely consumed by two more reports on the controversial writing portfolio program. Both presenters are portfolio proponents. They exhausted the remaining time in the second half of the meeting without allowing any time for questions, which didn’t seem to sit well with many task force members. Sadly, both presenters had good handouts. These could have saved a lot of presentation time had they been sent to the committee members prior to the actual meeting.

Anyway, perhaps as evidence of the committee’s displeasure, an attempt to see if there was any sort of consensus on portfolios was quickly shot down. Instead, the presenters were invited to the next meeting to answer questions.

Thus, after four three-hour-long meetings, task force agreement on writing remains as elusive as ever.

Even worse, some important information concerning portfolios still has not been presented to the committee. The first set of still-to-be-discussed evidence is from the National Assessment of Educational Progress Eighth Grade Writing Assessments. The Bluegrass Institute has written about this subject, and we see Kentucky’s rather poor performance on NAEP writing as evidence that teachers in other states are teaching writing – much more effectively – without portfolios in their state accountability systems. This national assessment material was requested by Steve Stevens, head of the Northern Kentucky Chamber of Commerce, at task force meeting two. As of the close of meeting four, it still has not been presented.

The second set of evidence concerns a writing portfolio audit for the past year. The audit for 2007-2007 was presented at meeting three, and the 2007-2008 audit was promised for this meeting. It was never mentioned.

Thus, at the close of meeting four, only two major issues, portfolios and arts/humanities assessments, have been discussed in any detail. With a host of other items on the agenda still to be considered – including: four separate items on general assessments, standards, longitudinal testing models (a 1998 legal requirement that has never been addressed), changing CATS focus to individual students, college readiness issues, analysis of the EPAS system, balance of school/student accountability, timeliness of results to schools, and end of course exams – it seems clear that this committee is now well behind. With only two meetings scheduled in October, and maybe one or two more, still unscheduled, for November, the committee is at least half way through with little to show for all the time expended to date.

2 comments:

Anonymous said...

Where are the reliability and validity statistics? Are there any?

Anonymous said...

RE: Anonymous 8:04's Question

I assume you are referring to validity and reliability statistics for the writing portfolios.

The best data I have seen is found in an audit of the scoring accuracy of portfolios that was completed on the 2006-2007 year’s submissions. Unfortunately, I don’t believe the KDE ever put that on line (dirty laundry, you know). The Family Foundation of Kentucky forced this into the public area with an open records request last year.

The audit title is “Kentucky Commonwealth Accountability Testing System, 2006-2007 Writing Portfolio Audit Report.” You might search the KDE Web site to see if this is now available. Otherwise, you will have to write to the KDE, 500 Mero Street, Frankfort, KY 40601 to request a copy.

If you do get a copy, go to pages 14 to 17 where you will find summary tables showing the percentages of exact agreement between original scores and the audit determined scores. Pay particular attention to the percentages originally scored “Proficient” and “Distinguished” versus the score distributions that resulted when the audit was conducted.

For example, in the “All Grades” combined summary, of 588 portfolios originally graded “Distinguished,” the audit said only 37 -- yup, just 37 -- actually deserved that score.

Of the 4,483 portfolios originally graded "Proficient" 18 kids were short-changed and should have gotten a "Distinguished" score. However, 2,721 students -- 61 percent -- should have gotten scores less than "Proficient." That is a huge error, especially when you consider that proficiency is supposed to be the goal of CATS.