More on the (probably) faked Civics survey done for G.I.

by David Safier

A few days ago, I posted about problems with the survey Goldwater Institute's Matthew Ladner used for his study about how little high school students know about Civics. It seems the company that conducted the survey, Strategic Vision LLC, has a history of complaints about their methodology and their failure to supply documentation of their work. FiveThirtyEight.com analyzed the survey using some sophisticated statistical formulation and found that the Civics survey, as well as many of their electoral polling surveys, appear to be fabricated. (You'll find a long list of links about the company at the end of my earlier post.)

Since then, Adam Molnar, an assistant prof at Bellarmine University in Kentucky who teaches statistics, emailed me, saying he had done something a few of us here were talking about. He gave the identical civics test to his students using the same basic technique used in the survey. Then he did something interesting. He focused his attention on the wrong answers he got from his students and compared them with the wrong answers on the Strategic Vision survey. He found his students gave a much wider range of wrong answers, which adds to the suspicion that Strategic Vision made up the wrong answers on its survey, which also means it made up the survey.

Molnar has a small sample of 41 students, so he's not claiming he has statistically accurate results. But he reasons that, if his small sample had a wider range of wrong answers than a survey with 2,700 students — over 60 times as many — something is wrong with the larger sample.

Some examples:

A question asks, "How many justices are on the Supreme Court?" Not one of the 1,350 private school students in the AZ survey gave the right answer, 9. Oddly, their only answers were 6, 8, 10 and 15. Public school students answered 7, 8, 9, 10 and 12. Molnar's students answered 3, 6, 7, 8, 9, 12, 13, 31 and 39. His students have a more believable distribution.

He found the same situation with the question, "We elect a Senator for how many years?" AZ private school students limited their answers to 2, 4, 6, 8, and 10 — all even numbers, implying every student knew a Senator didn't serve an odd number of years. Public school students gave the same 5 answers. Molnar's students, on the other hand, answered 2, 4, 5, 6, 7, 8, 9, 16 and Life.

I don't think there's much chance college students would have a wider range of wrong answers than high school students, but that's what Molnar found. His conclusion, and mine: Strategic Vision made a best guess about how students would answer without going through the bother of actually conducting a survey.

I've posted the paper Molnar sent me about his findings after the jump.