More on using tests to rate teachers

by David Safier

Commenters have been doing my work for me lately — and better than I do it in many instances. Here's a terrific comment by davewave64 on the subject of rating teachers based on student test scores. As you'll see, he's in the ed research profession. My favorite idea: his conclusion that one unintended consequence of rating teachers based on test scores would be, the grades and subjects where students aren't tested would get an infusion of the least competent teachers while the best teachers might flock to the tested grades and subjects.

Here is the part of his comment he says is "an excerpt of something I wrote recently for the folks who are helping the State of Arizona with their 'Race to the Top' application."

I’ve worked in a large urban district's research department for more than 17 years. During my tenure with the District, I’ve been involved in numerous studies that linked student achievement to individual teachers. None of that work ever made it out of the experimental phase into an implementation where the results were used in teacher evaluations, or decisions about pay and employment. Without an external impetus, political pressure within the district made it impossible to implement such a program. However, we’ve done the calculations enough times to know some of the pitfalls to measuring “teacher effectiveness.”

First, there is a data coverage problem. We currently administer the AIMS reading and math tests in grades 3 through 8 and one or more times in high school. Arizona will be reducing the Writing portion of the test this year to just grades 5, 6, 7 and high school. And, science is administered just three times (4, 8, and high school). Teachers who teacher in off grades and subjects do not have an achievement measure. With changing teacher assignments, staff and student mobility between districts and among district schools, and the fact that high school students are not required to re-test if they pass the AIMS on their first attempt in the spring of 10th grade, even teachers who have taught grades and subjects that are assessed may not have consistent, reliable data for use in the growth calculation. When you consider these limitations, you may have as many as half of your teachers without sufficient achievement measures.

Second, the lack of data coverage is not uniform across grades. Early childhood grades, which are some of the most crucial years, have no coverage. And, despite their complex array of subjects/course difficulty levels, high schools have only one measure of reading, writing, math and (life) science achievement for 60% or more of their students (most students pass the subject tests the first time and do not retest). Only middle schools have near complete coverage across their grade span. And it is our middle schools that have the greatest difficulty finding Highly Qualified teachers. Disproportionately emphasizing “teacher effectiveness” at the middle level may exacerbate the current teacher shortages in those grades.

Third, there are limitations to the “power” of the resulting effectiveness measure. When a measure like this is used to determine employment status and pay, you need to be pretty certain that the results are objective (i.e., accurate and fair). But, in practice, what you are creating is essentially a rank order of your teachers. The best teachers rise to the top of the list and the worst settle to the bottom. Where to draw the “effective” and “ineffective” lines is more an art than a science. What we’ve found is that conservatively you can identify the upper 20% and the lower 20%, with 60% of your teachers left in the middle, or average range. So, in the end, you have “actionable” information on perhaps 20% of your entire teacher workforce (i.e., 40% of the 50% who have data). And, unfortunately, given the current AIMS administration dates, effectiveness ratings would not be available until June or July, well past the customary evaluation, termination, and hiring deadlines.

Over the years, we’ve consulted with a small number of principals who we consider to be our best administrators. At various times we’ve asked them to give us a subjective ranking of their teachers that we could compare to our objective effectiveness rating. We’ve generally found a high correlation between their subjective rankings and our objective rankings. So I think that principals would appreciate having an objective measure to back up their observational data during a teacher termination process. I also think that our best teachers would appreciate “merit pay” and would support the termination of the worst teachers. However, I’m not sure if these obvious benefits will outweigh the unintended consequences of such a system.

As I’ve pointed out, the unbalanced coverage of assessments by grade and subject will place a disproportionate emphasis on middle schools, and intermediate grades. Weak teachers in “covered” grades may request transfers to grades/subjects not well covered by assessments. Strong teachers, seeking increased pay might request transfers to well covered grades. Unless there is an equally robust “observational” component to the teacher evaluation process we will simply churn our existing teachers from one grade level to another without substantively increasing the overall quality of instruction. In fact, we could unintentionally reduce the overall quality of instruction as teachers move to new assignments to jockey for position in the system (whether avoiding sanction, or gaining reward).

The matching of student data to individual teachers is a simple task in comparison to the organizational work required to make good use of the resulting measures. In my experience, there is no substitute for effective supervision.

…snip

So, if you had a child in a classroom with an ineffective teachers, would you rather have an effective principal who is trained in evaluation and who can fire an ineffective teacher, or a data system that will tell you six months after the year is over that the teacher was ineffective? I say we spend the 4 billion dollars on training better Principals and then giving them greater freedom to fire teachers.

Or, would that be too much like Charter schools?