Taxpayers win as state policy “research” center sunsets

The Kentucky Long Term Policy Research Center (KLTPRC) closed its doors for the last time today. All of its funding was cut in the new state budget that goes into effect tomorrow.

While I fully appreciate the desirability of well done and insightful policy studies, it’s no secret that I have been highly critical of one set of KLTPRC reports on education. Those rank all the states on a KLTPRC-contrived ‘Education Index.’ Aside from providing a very dubious ranking system, this work basically duplicates effort that should, and does, come from another legislative agency, the Kentucky Office of Education Accountability (OEA). OEA has more resources and talent in this area and didn’t need the attempted upstaging by the KLTPRC.

Thus, at least in the education area, it looks like the KLTPRC’s demise is a win for the taxpayer.



The KLTPRC’s dubious education ranking series was published as a string of ‘Policy Notes’ starting with Policy Notes #23 in October of 2007 and ending in May 2010 with Policy Notes #34.

These Policy Notes contain many statistical problems, which became much worse in our view after we notified the KLTPRC of our concerns and the agency blatantly refused to modify their index scheme.

That refusal is hard to defend. Two of the Index items should have been promptly eliminated as soon as we pointed them out.

The first unacceptable data ranking compared Kentucky’s high school dropout data against other states. The serious problem with that is that Kentucky's dropout data was officially audited by the Kentucky Auditor of Public Accounts in 2006 and found to be inaccurate.

Worse, the KLTPRC continued right on ranking these unreliable dropout numbers more than a year after we discussed both the audit and the extensive general research on the problems of state-reported high school dropout rates with the KLTPRC’s education researcher.

The second unacceptable data ranking was for the ACT college entrance test. The KLTPRC ‘Index” ranks all the states’ performance on the ACT. This is not statistically defensible because participation of high school graduates on the ACT varies dramatically from state to state.

For example, in 2009 Kentucky had 100 percent participation on the ACT while in Maine only nine percent of the high school graduates took this test (most in Maine take the SAT).

Even the ACT, Incorporated, which creates this test, specifically told us they strongly discourage doing such rankings because participation varies so widely from state to state.

But, that didn’t stop the KLTPRC, not even after we brought these serious statistical issues to their researcher’s attention over a year ago.

Then, there is some classic ‘apples to oranges’ “stuff” in the Policy Notes.

The KLTPRC uses four-years-out-of-date 2005 National Assessment of Educational Progress (NAEP) science data for its 2009 calculations. However, KLTPRC doesn’t include any NAEP science scores in the 1992 comparison calculations.

Of course, if the seriously out-of-date science scores were not included in the KLTPRC’s 2009 Education Index, Kentucky’s ranking would be notably lower.

A somewhat similar problem shows up with the inclusion of NAEP eighth grade reading scores. They are included in the 2009 Index calculation but not in the one for 1992 (which was six years before the first administration of this particular test).

Including data in one year but not in the other year is a classic abuse of statistics. Actually, these aren’t even ‘apples to oranges’ mistakes – it’s more like comparing apples to mirages.

There is a lot more wrong with the simplistic comparison of NAEP scores in the KLTPRC’s Index. The entire process ignores the serious NAEP interpretation issues that have been created by dramatic increases in minority populations in many other states. I’ve written a lot about NAEP interpretation problems in the blog (use the search term “NAEP” in the blog’s search tool), and you can also find a nice discussion, with examples, in this freedomkentucky.org Wiki item.

Again, KLTPRC turned a deaf ear to us when we pointed out these NAEP issues last year.

I must stress that problems with its education studies does not necessarily mean that other KLTPRC projects had similar issues. However, the obviously poor quality of the KLTPRC’s education work doesn’t raise confidence, either.