As you know if you've been reading this blog, I've had several bones to pick with the Goldwater Institute's studies comparing public and private school students in their knowledge of civics, tolerance of others and feelings about the schools they attend. (My criticism of the Civics study is here, and the criticism of the other two are combined in this post.) But here's one objection that never occurred to me until two faithful readers emailed me material that puts the polls themselves into question. (Hat tip to todd and to Eli Blake.)

It's possible the polling company, Strategic Vision LLC, simply made up the numbers in the surveys it gave to Matthew Ladner at G.I. which formed the basis of the three studies.

Nate Silver at FiveThirtyEight believes Strategic Vision LLC concocts fraudulent surveys on a regular basis. In case you haven't heard of Silver, . . . If you're looking for the smartest guy in the room — more specifically, in any room where statistics are the topic — Ladner and I would be hanging around in dark corners looking confused and Silver would be standing in the spotlight with other stat heads gathered around listening to what he had to say.

In one of his many posts on Strategic Vision (I list all I could find at the end of the post), Silver looks at a survey Strategic Vision conducted in Oklahoma which asked the same 10 civics questions G.I. asked students in Arizona. It got equally dismal results, indicating that Oklahoma students, like those in Arizona, know very little about civics. Ladner was involved in setting up the Oklahoma survey and wrote about the results on the Oklahoma Council of Public Affairs website.

Funny thing, though. Oklahoma state Rep. Ed Cannaday asked the same 10 questions of high school students using the same basic methodology, and his students appeared to be about 3 times more knowledgeable than the ones Strategic Vision surveyed. Cannaday, by the way, is a former teacher and school principal, so he knows a bit about education.

[Cannaday] arranged to have all the seniors in the 10 secondary schools in his district take the Strategic Vision/OCPA survey. Cannaday tried to replicate the Strategic Vision survey to the greatest extent possible. The same exact questions were used, and as in the case of the original survey, the answers were open-ended rather than multiple choice. The survey was administered to a total of 325 seniors, including special education students.

Ladner has responded to the allegations that Strategic Visions made up the numbers, saying he will look into them. "If I got snookered," he wrote, "I’ll own up to it, but the jury is still out."

As it happens, about 3 weeks ago, I asked Ladner if I could see copies of the actual Arizona survey results, and he was kind enough to fax me the two surveys used for all three Arizona studies — one survey of public school students and another survey of private school students.

I'll do some of my own analysis of questionable aspects of the surveys after the jump. And at the end of this post, you'll find a long list of links from a number of sites concerning Strategic Vision's questionable integrity.