High stakes, low value tests

by David Safier

The nation bought into high stakes testing for accountability and incentives with no evidence they do any good because, well, because the idea sounds like it makes so much sense. Test the students, then improve their performance, then reward teachers and administrators whose students make the most progress. What could be simpler?

The problem is, it doesn't work so well in the real world. From Education Week (subscription only):

A blue-ribbon committee of the National Academies’ National Research Council undertook a nearly decade-long study of test-based incentive systems, including the “adequate yearly progress” measures under the No Child Left Behind Act, high school exit exams, teacher merit-pay programs, and other testing-and-accountability initiatives. While the panel says it supports evaluating education systems and holding them accountable, on the whole it found the approaches implemented so far have had little or no effect on actual student learning, and in some cases have run counter to their intended purposes.

Educators who have objected to the overreliance on high stakes tests for years feel vindicated by the results. But even conservative educator theorists like Eric Hanushek are taking the study seriously.

Eric A. Hanushek, an economics professor at Stanford University, said he was “stunned at how broad” the findings were. But he warned against using the committee’s critique of test-based incentives to throw out accountability systems in education altogether.

“Some form of accountability is undoubtedly useful, but you have to be careful with how you structure accountability systems,” Mr. Hanushek said. “What we’ve done to date hasn’t been perfect; there are lots of obvious flaws in either results or program structure to date. As we go into the future, we should learn from our results.”

Simpler Hanushek: We've screwed up testing so far, but I'm sure we can figure out how to make it work.

An obvious problem is, if teachers and students know their futures will be determined by test scores, they spend time figuring out how to game the tests rather than learning.

[E]ducators facing accountability sanctions tend to focus on actions that improve test scores, such as teaching test-taking strategies or drilling students closest to meeting proficiency cutoffs, rather than improving learning. Such a response undercuts the tests’ validity, the report says.

As an example, the report points to New York’s requirement that all high school seniors pass the state regents’ exam before graduating from high school. The policy led to more students passing the tests, but scores on the lower-stakes National Assessment of Educational Progress, which was testing the same subjects, didn’t budge during the same time period.

The sad conclusion:

[S]chool-based accountability mechanisms under the NCLB law have generated minimal improvement in academic learning, the study concludes. When the systems are evaluated—not using the high-stakes tests subject to inflation, but using instead outside tests, such as NAEP—student-achievement gains dwindle to about .08 of a standard deviation on average, mostly clustered in elementary-grade mathematics.

NOTE #1: I'm not anti-test, and neither is the study. Over my 30+ years of teaching, I made my students eat their peck of tests, believe me. But I never believed tests told me everything there was to know about my students, which is what our current, NCLB-fueled testing frenzy maintains. Tests are an imperfect assessment measurement, one of many tools in an educator's toolbox to help determine what students know and what they don't know.

NOTE #2: The National Academies began in 2002 and is "chartered by Congress to provide policy advice on science, technology, and health." I mention that because I hadn't heard the name before, so I imagine most readers probably don't know any more about the group than I do.