One reason (of many) you shouldn’t trust Florida achievement scores

by David Safier

As Goldwater Institute’s Matthew Ladner never tires of pointing out, Florida’s 4th grade reading scores on the NAEP, the national achievement test which is about the best thing we have to compare state achievement, have improved faster than most states since Florida embarked on its reforms in 2002. But lots of people, and I am among them, take those scores with many grains of salt.

The lastest skeptic is Madhabi Chatterji, from the Teachers College, Columbia University, who reviewed Ladner’s latest paean to the “Florida Educational Miracle” and found it wanting. The main reason is, Florida’s policy of 3rd grade retention skews 4th grade reading scores. The lowest scoring readers are held back, so they take the test when they are one year older. Chatterji likens it to comparing the heights of students in the 4th grade when one state holds its shortest students back a year. When they’re measured with other 4th graders, they’re a year older and hence a year taller. It makes any state-to-state comparison nonsense.

Arizona is going to start 3rd grade retention as well. Be ready in 2-3 years to hear how our NAEP test scores for 4th graders have improved, using the same flawed reasoning.

I decided to chart the number of students in each grade in Florida since 3rd grade retention began in 2002. Here’s the graph for the 2004-5 school year.

Note how third grade bumps up because of the students who have been retained, and 4th grade is 5% lower. The reasons get complicated, but basically, the 4th grade size is a combination of more 3rd graders being held back and fewer 4th graders being held back (In Florida, students can be held back in any year). Combine those factors, and Florida’s 4th grade classes are an achievement-rich environment, where the weakest readers are a year older than other students, and at the end of the 4th grade year, a larger-than-normal number of its weakest students are passed on to the 5th grade.

I looked at the number of students in each grade in other states, and few if any had such a big drop from 3rd to 4th grade.

You probably noticed that I circled the huge dropoff from 9th to 10th grade — a whopping 19%. The reason for the drop? Passing Florida’s 10th grade state FCAT test, the equivalent of Arizona’s AIMS, is a graduation requirement. Florida bumps up its percentages of FCAT passers in the 10th grade by keeping lots of students in the 9th grade. What happens to them after that is anyone’s guess, but if you look at the big slide in the 11th and 12th grade, it can’t be good. Florida has one of the worst graduation rates in the country. The same thing happens in other states from 9th to 10th grade, including in Arizona, but the only other state I found with that big a drop-off was Texas, which is notorious for boosting its 10th grade state test passage rate using a number of student-unfriendly methods.

Here is a chart of the sizes of each grade in Arizona in 2004-5 for comparison.

Arizona has a 2% drop-off from 3rd to 4th grade. Most years, it averages 1-2%, far smaller than Florida’s average 4-5%. No wonder Arizona’s 4th grade NAEP scores haven’t had the artificial inflation of Florida’s. And the drop from 9th to 10th grade is 10%, about half that in Florida’s.

This is one of many reasons that the “Florida Educational Miracle,” which Matthew Ladner and Jeb Bush are going around the country selling, is 95% snake oil and 5% truth. But unless you’ve developed a Ladner-specific crap detector, as I have, which tells you that most of what Ladner says is crap (Remember his famous “Bus drivers are bureaucrats” defense, which he still maintains is true?), you’re likely to accept his impressive-but-bogus statistics as truth.