The Floridation of Arizona Education: Reading scores

by David Safier

 Saturday I offered a brief preview of some posts I've planned looking at the "Florida educational miracle" that Arizona's conservative educators are hell bent on duplicating — at least the parts they like. The most important piece of Florida-based legislation this year is one to hold back 3rd graders with low reading scores.

The Goldwater Institute's Matthew Ladner has been touting Florida's educational reforms for years and is the prime mover in the attempts to Floridate Arizona. His favorite mantra is, "Florida's 4th grade test scores have improved while Arizona's haven't." I want to look at that assertion in more detail.

First, Ladner is right about the numbers. Florida's 4th grade reading scores have shown significant improvement in recent years, and the increase of African American and Hispanic students' scores has been especially dramatic. During this same period, Arizona's scores have remained pretty stagnant. Ladner is using scores on the national NAEP test, which is considered the gold standard in national comparative testing, so they are as valid as any you're going to find.

But are the increases a reflection of Florida's reforms?

In 2002, Florida began holding back about 10% of 3rd grade students who had the lowest reading scores (Florida enacted other reforms as well, but for the sake of simplicity, I'm zeroing in on this one). That means during that first year, the lowest scoring students didn't take the 4th grade test. Naturally, average test scores rose. It would be like taking the lowest 10% of U.S. incomes out of a study of income. The average income in the study would be higher simply because the lowest incomes were not included.

The next year, the group which was held back entered the 4th grade and took the test alongside other 4th graders. But they were a year older than they would normally be when they take the test — approximately 10 years old instead of 9. They were the age of 5th graders, but they were taking a test as 4th graders. One extra year of life is very significant at their young age, and the extra year of education meant they had been in school something like 25% longer. Both of those factors would mean they would tend to be better readers, and you would expect their reading scores to increase.

In other words, a good part of the boost in reading scores since 3rd graders were held back can be attributed to the extra year of age and education of the lowest readers, not to an increase in achievement based on educational reforms.

Ladner also uses the 4th grade scores to show that African American and Hispanic students had an especially strong boost in reading scores, which is true. But he doesn't mention that something like 15-20% of the African American and Hispanic students were retained as compared to only 4-6% of the Anglo students. Naturally, the more students in a group that get an extra year before they take the 4th grade test, the more that group's score will improve.

I'm not surprised Ladner doesn't deal with these problems in his studies. As a propagandist, he's far more interested in pointing out the results that push his agenda than trying to offer an honest assessment of the data.

Unlike Ladner, I'm going do my best to be fair and objective in my analysis.

It may be that students genuinely became better readers because they were held back in the 3rd grade and that the change wasn't simply based on their being one year older and having one more year of schooling. The question is, how can we separate the extra year in age and schooling from any genuine academic increase due to their being held back?

Here is what I might do if I were a researcher with access to Florida's education data. I might take the students who weren't held back a year and see how much their reading scores increased from the 4th to the 5th grade. I would compare that with the increase in reading scores of the retained student from their second year in the 3rd grade to their year in the 4th grade. If you subtracted the increase of the students who weren't retained from the increase in those who were, the result should show how much extra academic value the retained students got from repeating the 3rd grade.

(To make the comparison even more significant, a researcher could only use the increases of the lowest 10% of students who weren't retained, since poor readers tend to advance more slowly than proficient readers.)

Unfortunately, you can't use the NAEP test for the comparison I suggest, since it's only given in the 4th and 8th grades. But all students take Florida's FCAT tests yearly. Using FCAT scores wouldn't create a perfect comparison, but the FCATs would be an adequate proxy to eliminate the amount of the increase that is purely a function of the one extra year in age and schooling.

While Ladner jumps through all kinds of other hoops to create a false impression that he's a thorough researcher who is trying to deal with all possible factors, he doesn't take the extra step with this vital variable, so his conclusions about reading increases due to student retention is skewed, and basically worthless.

Let me look at two more factors that add even more questions regarding the direct effects of Florida's reforms on student achievement.

The table at right shows the increase in the percent of students scoring at basic and above in 4th grade reading. The dark line is for Florida, and the lighter line is the national scores.

The Florida scores showed their steepest increase from 1994 to 2002, yet Florida's educational reforms didn't begin until 2002. After the reforms were put in place, the slope flattens out a bit.

If I had nothing but the chart as evidence of 4th grade Florida's reading scores, I would ask myself what kind of amazing reforms were put in place between 1994 and 2002 to create such a major spike in reading achievement (Answer: none that I know of) and what caused that increase to slow after 2002, after the "Florida educational miracle" began.

To be fair (because that's the kind of guy I am), I would also note that Florida's scores continued to increase while the national scores stayed flat, which shows Florida's improvement was better than the  national average. But no matter how you slice it, Florida's most dramatic increases came in the 8 years before the reforms kicked into gear. That puts the cause and effect — reforms resulted in reading increases — further into question.

Finally, whatever student reading gains occurred in the 4th grade, they seem to have shrunk considerably by the time the students reached the 8th grade, according to a table Ladner uses in his study. If that's true, then it's not accurate to judge Florida's overall academic improvement by 4th grade scores, since what matters is the level of competence students have achieved by the time they leave school. If earlier gains don't result in later gains, their value is questionable.

To be fair (because that's the kind of guy I am), any gain in 8th grade scores due to student retention in the 3rd grade wouldn't begin until around 2006 when the first batch of 4th graders became 8th graders. But for reasons that escape me, Ladner's sets up comparison between 1998 and 2007 scores. Why his starting point is 4 years before the reforms began is beyond me. Isn't the purpose of his study to show how Florida educational reforms resulted in "The Florida Miracle"? My unsubstantiated suspicion is, 1998 was the starting place which resulted in the most dramatic increase.

I've mainly dealt with 3rd graders being held back in this post and disregarded other educational reforms because our legislature is focusing on holding 3rd graders back. There are other aspects to the reform, and more data about Florida education. I'll take those up in later posts.