The data-driven education dilemma

by David Safier
OK, this is a tough one. There are so many on-the-other-hands here, I'm running out of hands.

The Obama administration took aim on Thursday at state laws — adopted after heavy teachers’ union lobbying — barring the use of student achievement data to evaluate teacher performance.

The federal Department of Education proposed rules to prevent states with such laws from getting money from a $4.3 billion-educational innovation fund.

Obama and Ed Sec Arne Duncan are big believers in collecting data on students and using it to evaluate student achievement and teacher performance.

On the one hand, we all want student achievement to improve, and we want to keep and reward our best teachers while we weed out the incompetents. On the other hand, increasing our emphasis on data collection may not be the best way to accomplish those ends and may make things worse, not better.

Here's how data-driven education works. You check each student's achievement as often as you can, trying to diagnose weak areas that need to be improved and looking for overall growth. The idea is to measure each child's progress for the purpose of accelerating that progress, and see how a class full of students progresses to assess each teacher's competence.

So if Child A is two years above grade level in reading in the 3rd grade, then slips to only one year above by the 5th grade, you look carefully to see if there's a problem, with the child, the teachers or both. On the other hand, if Child B is two years below grade level in the 3rd grade, then jumps to one year above by the 5th grade, you break out ice cream and the champagne and throw a party.

If a class of students raise more than a grade level in a given year, that's probably a good sign that the teacher is competent. If the students rate of growth slows, that's a warning sign that a teacher either needs help, or needs to find another line of work.

All that sounds good. On the other hand . . .

The only somewhat objective way we have to measure student achievement is through standardized tests, like AIMS, NAEP, etc. And those are very, very crude measures. Real education is filled with intangibles, and, by definition, you can't test intangibles. In reading, for instance, you can more-or-less measure if students can decode written language and comprehend its meaning in short passages, but that's about all you can measure about students' reading ability. Other factors that involve deeper, more personal understanding or joy of reading, both of which create lifelong readers, can't be measured in any objective way.

And tests aren't even a reliable measure of the tangibles. As we all know — teachers more than anyone — you can effectively raise students' scores by teaching to the test. What you get are better testers, not better educated students.

And of course, you can't test art, because art is untestable by any objective measure. And it's difficult to test history and science, since they're more knowledge and concept based than skill based. So we end up focusing more and more time on the rudiments of reading, writing and math to get those scores up, to the detriment of other subjects.

That sounds bad. On the other hand . . .

If a child never learns to read fluently, or write coherently, or perform basic math competently, most of the other things you want to teach that child aren't going to sink in very well. That person is most likely doomed to low wage jobs or worse and will go through life with a very limited understanding of what goes on in the world, no matter how well rounded the school's curriculum is.

Much as it pains me to say it, if someone convinced me education is a zero sum game, and there is a way to teach reading, writing and math successfully to the lowest achieving students, but it would mean slighting the rest of their education, I would say, "If that's the only way to do it, go ahead. Do it!"

So if a data-driven educational program succeeded at genuinely adding, say, two grade levels to the lowest students' achievement scores — not just by improving testsmanship, but by genuinely improving skills — I would reluctantly applaud the program even if it slighted other areas of the curriculum.

On the other hand . . .

I have yet to see those magic bullets that go to the heart of a student's reading, writing and math skills and lift them to significantly greater heights. So far, what we've managed to do is drill the students more, test them more, then start prepping them for the next set of tests without seeing much in the way of results. All we've gotten for this data-driven, skill-based education is a more rigid educational system that slights those wonderful aspects of our education that can't be tested.

It may be that the best way to increase students' basic skills is to teach a fuller, richer curriculum in a way that encourages students to want to learn. Or there may be some magical, mysterious combination of skill based and conceptually based learning, where one feeds on the other. However, I have yet to see that type of magic bullet either.

To sum up . . . On the one hand, I don't want to keep things as they are. That's unacceptable. But on the other hand, I don't want to replace the status quo with status worse.

Have I ever told you how tough education is, and how difficult it can be to figure out how to make things better?

0 responses to “The data-driven education dilemma

  1. David Safier

    Terrific analysis, David (No, I’m not patting myself on the back. The other David). You state the concerns extremely well.

    Number crunchers are also cost-benefit analysis people, or they should be. And as David says, the cost of the phenomenal amount of data that has to be collected, sorted and analyzed — trust me, it will go far beyond test scores if these people have their way — will be huge. It will take a whole new bureaucracy and cost tens of billions, at least, nationwide. Will we benefit enough from that data to make it worthwhile? Could we spend the money more wisely?

    I don’t know if this is a real term, but there needs to be a harm-benefit analysis as well. Will the benefits of a data-driven education significantly outweigh the harm? I have my doubts.

    I say all this knowing that test scores and other data are going to become a growing part of education. The proponents have to be shown the risks involved so they don’t rush headlong into something that we later regret.

  2. David Scott

    David, et. al., I guess I should have disclosed the fact that I am a number cruncher for a large public school district. Francine, I appreciate the value of data, but I share David’s concerns about data collectors. Do we really want to give Tom Horne the ability to name the best and worst teachers? I certainly don’t.

    As I pointed out, the amount of data available for calculating teacher effectiveness is fairly limited at this point. We will need to spend a whole lot more money testing more kids more often before we will get to the point were we can fairly evaluate all teachers. We will also have to spend a considerable amount of money for data systems to share relevant data between schools, and State and National agencies.

    In the end, we may actually be able to say who is and isn’t a good teachers. But at what cost, both monetarily and politically? Education is a local issue. Politically, this data movement will strip local agencies of control – which may be a good thing in some cases. But, before we go merrily down this path, we should understand the consequences of our actions.

    This issue may actually bridge the liberal-conservative divide. Conservatives will complain about the expansion of “big government.” And, liberals will cry foul over the narrowing of instruction as teachers feel greater pressure to “teach to the test.”

    Unfortunately, the real losers in this game are our kids. As grown-ups continue to argue about what’s wrong with schools, and fail to fix them, kids (Arizona kids in particular) continue to attend sub-standard schools.

    We actually have the tools today, to determine which teachers are effective. The most important aspect of this process is an effective principal. Good principals are in classrooms everday, and know which teachers are doing great in their jobs. Good principals are also able to provide professional development to teachers who may be good, but could be better. And, finally, good principals do the hard work necessary to fire bad teachers.

    The biggest problem with this system is not lack of data, but lack of money. Good principals aren’t hard to find, but they’re hard to sustain. With limited budgets, and low pay, we just aren’t getting the best teachers entering the profession. And, doing the work necessary to fire a teacher is unrewarding and can take an emotional toll on the principal. Many principals will turn a blind eye towards a weak teacher, because they know they may end up with a worse teacher as a replacement.

    A data-driven teacher evaluation system, while de-humanizing, would certainly be more objective in targeting teachers who need to leave the classroom. And, if school districts adopt policies to fire teachers based on the system, principals would nolonger have to do all of the messy work to fire someone. But, without higher salaries to attract better teachers, we’re just going to churn more teachers in and out of our schools.

    Given the amount of new money needed to test kids, track data, and compute teacher effectiveness, I can’t help but wonder if that money wouldn’t be better spent on higher teacher salaries. Additionally, given the number of years of data necessary to determine an effective teachers, isn’t the principal’s real-time evaluation and teacher-feedback more beneficial to kids?

    No data-driven teacher effectiveness program will ever give teachers immediate feedback on their classroom management, student engagement, and use of multiple perspectives in delivering a lesson plan. And, no amount of number crunching will produce a professional growth plan will help a struggling teacher to improve.

  3. Also, following up what I wrote a moment ago,

    In Germany and other countries they test students fairly early and divide them into groups which are college bound and trades-bound.

    Obviously in the U.S. this could be a huge can of worms for the reasons I alluded to at the end of the last post, but at the same time what we have now is a system in which we teach some things which some of the students will never need (because in effect our whole basic curriculum is designed to prepare people to go to college) which only causes more of them to drop out (and either end up destitute or learn the trade they wanted to learn in the first place someplace outside of school) and at the same time because we do try to cram even students who have no need for some of these courses into them we waste resources and our college bound seniors in many cases still need serious remedial work when they get there.

    As I said, you’d have to be very careful to avoid anything that smacks of racial or gender bias but in many cases the students know themselves what they want to go into and maybe we should let them do more of it in college (i.e. let a budding young mechanic intern in an automotive repair shop instead of taking another semester of algebra.)

  4. David, Francine et al,

    You miss out on the most basic variable: the students.

    My observation is that some teachers work better with specific students and some who may work well with one student fail miserably with another.

    I had a professor in college (and graduate school) who was among the best physical chemists in the world, literally on the frontiers of his field. I thought he was also quite possibly the worst professor I’ve ever had– although he was enthusiastic he’d get lost on tangents and get into topics that were so deep that if you understood what he was saying it was probably literally true that at that moment you and he were the only ones on the planet who did comprehend it. Needless to say I learned very little from him, and most of what I learned was not the course material. It wasn’t just my perception either. When we took our final exam one year one of the students brought a bottle of brandy (which the prof allowed us to pass around during the test, and it actually was fitting, I doubt if anyone got many questions right but he was nice and passed us anyway.)

    But then some years back my perception was altered when I read online about another of his former students who had made some major advances in physics and credited this professor with giving him the theoretical background to do it. In other words, if he didn’t teach the way he did, there would be things which are known today that would instead still be unknown.

    So was he a bad teacher? Perhaps, but beauty is in the eye of the beholder.

    No doubt there are indeed some teachers who either are completely ignorant of the material they teach (as a community college math instructor I see students all the time who have actually been instructed wrongly, likely by a H.S. math teacher who didn’t even have a degree in math– though that is a seperate issue because there just plain aren’t enough math majors) and also some who are so burned out that they just go into the classroom so they can pick up a paycheck and hate every moment of it and don’t give a flying leap when a student is having problems.

    But I’d also like to see a reform (and this is maybe just now becoming feasible) in which a student’s learning style is also tested and then matched to a teacher who is strong at teaching that type of student (instead of luck-of-the-draw like you have today.) But that could raise a whole new set of issues because you’d have to tread very, very carefully around issues of race, class and gender.

  5. Francine Shacter

    David, don’t knock data. Properly selected and correctly used, data have a definite value in understanding and improving -or wrecking – any system. Teachers are one component – that’s an undeniable fact. I don’t want to demonize teachers nor do I want to insulate them from participation in the education mix. Right now, anyone who “dares” to criticize teachers gets a united backlash from teacher’s unions. As I have said before, I had four children go through a very good and very homogeneous school system. Some of those teachers should have been canonized – they were terrific. Others? not so much. Like any other profession, there is variability in the quality. Denying it shifts the discussion onto the wrong foot and we’ll end up in an endless discussion that sheds much heat and very little light.

  6. David Safier

    David, I expect exactly what you predict: “a national student database for sharing data between schools and evaluating teachers at higher levels of government.”

    It worries me. Our obscene ability to collect and store data encourages the data collectors. But do they do us any good, or do they simply use the data to reach sometimes false, sometimes obvious conclusions about us while creating a detailed evidence trail of our lives?

    The most obsessed users of data are economists and insurance companies. Economists have made an art of getting it wrong to three decimal places, and insurance companies use the data they collect about us against us. The one thing the two groups have in common is that they believe they can reduce human behavior to numbers, and the people crunching those numbers usually have personalities at three removes from humanity.

    Do we want those same types of number crunchers to make educational decisions? Education is a truly human endeavor, where the entire person is the subject of the process. To let the kind of people who love to reduce the world to multilinear regressions have a major role in education — it worries me. We shouldn’t disregard the kind of data that can be reduced to numbers in education, but we embrace it at our peril.

  7. David Scott

    Perhaps the biggest problem with pay for performance is that currently there is not enough data to properly evaluate teacher effectiveness. In Arizona we test students with the AIMS in grades 3 through 8 and then again one or more time from 10 to 12, until they pass. We have also administered a string of nationaly normed tests (Stan 9, Terra Nova, Stan 10) in grades 2 through 9.

    It would seem that with this dataset we could fairly reliably evaluate the performance of our middle school language arts and math teachers, about half of our elementary teachers, and maybe half of our high school language arts and math teachers. But, when you factor in student turnover and teachers changing grades and subjects, the data begins to degrade. In the end, you will have four groups of teachers. You will have teachers who have sufficient data to be evaluated. In that group you will have a small number of teachers who are clearly superior to all others, and teachers who are clearly lacking. For the rest, the data just won’t have enough power to say much of anything about their performance. And you still have all of the other teachers who simply do not have enough data to be evaluated. So, the vast majority of teachers will not have a valid evaluation. And after all of the expense and time spent on crunching numbers, we already know who those clearly superior and clearly lacking teachers are.

    If we’re really going to do a good job of using student assessment data to evaluate teachers we need more testing, and better data systems for tracking the results. In fact, with Arne Duncan talking about developing national standards and a national assessment, don’t be surprised if he doesn’t start talking about a national student database for sharing data between schools and evaluating teachers at higher levels of government.

  8. I am all for accountability. For too long, bad schools were allowed to continue being bad. Excuses were made why the kids couldn’t learn. Lack of achievement was blamed on the student. NCLB changed that.
    That said, not all students enter school at the same level of achievement. The famous Hart and Risley study http://www.aft.org/pubs-reports/american_educator/spring2003/catastrophe.html
    shows that children from families on welfare, by age three, have heard 30 million words less that their peers from working class and professional families. Additionally, the child from the welfare family, hears twice as many discouragements as affirmations.
    No surprise then, the vocabulary level of the children at age three was predictive of language test scores at third grade.
    The children in the study, were English speakers. English Language Learners obviously start school with a greater disadvantage, not knowing the language of instruction.

    So, not all Kindergardeners start school at the same language level. Intensive intervention is needed immediately, when the child enters school. (Actually the intervention should be long before Kinder.) Language intervention for these children must continue throughout the school years.

    Accountability is fine. But measurement must show growth from the level with which the student enters school. The playing field is not level. As David said, the students who do the best are those with the internal fire, who want to learn.

    Another major issue at schools with children from poverty is the mobility level. We turnover about 200 students each year. I have no control over the education that the new student to my school received at previous schools. A significant number of students at my school, have been to numerous schools before age ten. I have to take that child at the level he/she presents. While we do an excellent, academic job at my school, we are not magicians.

    I’m not making excuses for schools not performing. I’m just telling you that the demographics and instruction in Catalina Foothills, is very different than the demographics and instruction in a high poverty school. Some teachers are not cut out for the work in a Title 1 school. We do weed those people out, but it is hard to find excellent teachers who feel the calling to work in a school in a high poverty area. Staff who choose to work in Title 1 schools, should be paid more.
    (The TUSD salary schedule, regardless at which school you work, goes from year 1 pay of $32,960 to year 11 pay, $37,595. After eleven years a teacher makes $4,635 more than their starting salary. How pathetic is that?)

    Francine, I saw that piece on the News Hour last night, too. We used to hire monitors at our school to watch the children at recess. Last year, we hired young men with Parks and Rec Kidco experience to be “recreational monitors.” They organized games and activities at recess. I don’t know that organized play raised achievement, but we had far less discipline issues at recess.

  9. Francine Shacter

    David, find yourself a bright, energetic statistician who can pool several related variables and based on the findings of these co-related variables, I’ll bet you get a better understanding of what is going on. I don’t think it’s all the teacher, or all the home or all any thing. Rather, there is an interrelationship between significant factors. So, identify your factors, weight them, if you wish and then mayhaps you’ll have a better understanding of what is going and on and how to change/affect the environment to bring about the change you are looking for.

    Tonight, on the Lehrer program, there was a segment on the value of recess – I hope Steve Gall watched it. It showed, among other things, how recess, properly used, can be a really great learning tool – as well as a necessary way for kids to let off steam. We have got to let go of stereotypes of what works and begin looking at problems more creatively. That’s really the goal.

    I had four children go through the very good school system in Montgomery County, Maryland – in the Washington, D.C. suburbs – and I can tell you from that varied experience that some of the teachers were great and some should have never been let into the classroom and still others should have had their praises sung from the rooftops!! For the teacher’s union to refuse to concede that teachers are one of the variables is unrealistic and results in poor outcomes.

    I could go on and on and on – but that’s the nub of it.