In the interest of fairness (even [especially] to the Goldwater Institute)

by David Safier

I never pretend to be objective, but I try my best to be fair. I go after G.I. regularly — fairly, I believe — but when a suspicion turns out to be  wrong, I should say something. Which is what I'm doing here.

Matthew Ladner put together three policy briefs comparing public and private school students. I think the methodology used on the studies is bad, as I've written in earlier posts, but I suspected the surveys were inadequate to even reach the shaky conclusions Ladner reached. Ladner was good enough to send me the original surveys, and the raw data looks fine, even if the conclusions are still questionable in my book. So what we have here is a serious disagreement, which is OK by me.

For anyone interested, G.I. commissioned two surveys of 1,350 students each, one of private school students and the other of public school students. Except for some of the school information at the beginning, both surveys asked the same questions — 35 in all. Ladner used different questions from the surveys to form the basis of each of the three policy briefs.

For the record, the reason I was suspicious of the surveys is the wording used to describe the surveys in each of the three studies. They differed enough that I wanted to see what was going on.

• The Civics test study described the survey like this: "Strategic Vision, LLC conducted the poll of 1,350 Arizona public high school students on November 21-23, 2008. A separate sample of private school students was taken during the same period." That's very accurate.
• The study about student attitudes toward their school worded it this way: "In November 2008, the Goldwater Institute commissioned Strategic Vision to survey Arizona public and private high school students regarding their schools. The company conducted a poll of 1,350 Arizona public and private high school students on November 21-23, 2008." It's not clear if that's 1,350 of each or 1,350 total.
• The study about students' tolerance of others uses wording that is simply inaccurate: "The Goldwater Institute commissioned a private survey firm, Strategic Vision, to survey 1,350 Arizona high school students to help determine how well Arizona high schools promote civic values." That says 1,350 total, not 1,350 public and 1,350 private school students.

Hence, my suspicions and concerns about the nature of the data. And now those suspicions have been put to rest.