Fool’s Gold: the Goldwater Institute Civics Test

by David Safier
I posted yesterday about Matthew Ladner's poll showing that high school students don't know basic civics. I'm sure he's right, and I think schools should do better, but I gently accused him of playing "Ain't it awful?" with the data — taking a pot shot at public schools by hyping a genuine finding.

Today, no more Mister Nice Guy. The survey is crap, and Ladner's reporting of it is crappier. It's typical of the worst kind of G.I. "research," where you take your information, look for the results you want and publish those as if they are gold plated findings.

Let's look at a few quotes from Tuesday's G.I. daily email, authored by Ladner.

The report "reveals only 3.5 percent of Arizona high school students have learned the basic history, government and geography necessary to pass the U.S. Citizenship test."

Note the precise figure: 3.5 percent.

"We hired a firm to interview 1,140 Arizona high school students and ask 10 questions drawn at random from the exam given to applicants for United States citizenship."

So, over a thousand high school students were surveyed, and only 3.5 percent of them could pass a U.S. Citizenship test.

This morning, the Star wrote about the survey in its editorial. It was a telephone survey. That means the students who took the test had to be home and willing to take the survey. They sat at home, out of school, probably no more than half interested in what they were being asked, and answered questions they had no stake in answering thoughtfully or correctly.

How seriously did they take their answers? When asked who was the first president, some answered, George Bush, Bill Clinton and Barack Obama. Someone who has never met a teenager might think answers like that show the depth of their ignorance. Anyone who had spent five minutes with a teenager in the last year would know those are flippant, "I don't give a damn about your survey" answers. Matthew, no one — no one — believes Barack Obama is the first president of the U.S. Those students are putting you on.

What was the most popular answer to all the questions, according to the Star? "I don't know." That answer is a typical dodge. Any teacher knows the teenager is really saying, "I'm bored with this, and I don't want to think about it, and I'm not sure I know the right answer so I won't even try because I might embarrass myself. So get out of my face. Are we through yet?"

How different would the results have been if the students were chosen randomly, sat down in a room with pencil and paper and given some kind of incentive for answering correctly? I'm betting the scores would have been significantly higher,and nearly everyone would have put down "Washington" as the first president.

In the email, Ladner didn't mention it was a non-random phone survey. But he was perfectly happy to use a percentage-with-a-decimal-point number for the passage rate: 3.5%. That's simply intellectually dishonest. Not incorrect. Intellectually dishonest.

This is the kind of thing that infurates me about G.I. They have oodles of money, they do lots of research and studies, but when they come up with a result they like, no matter how questionable, they put it out there like it's nobel prize material. They never seem to ask themselves the kinds of questions any honest researcher or scholar asks: "Can I really use a precise figure like 3.5 percent on a non random survey conducted over the phone with disinterested teenagers? No, I'd better reveal my methodology and make my figures more approximate."

I began my Fool's Gold series when Ladner took his favorite $9,500 per student figure for Arizona, using expenses no one else includes, and compared it against other state's education spending figures, then concluded we were in the middle of the pack in per student educational funding. The assertion was so ridiculous, he eventually had to take it back. But he and G.I. continue to use the "shock and awe" method of releasing research findings. Take the most shocking numbers you can cook up, put them out there and stick with them.

You're basically a serious guy, Matthew, and as smart as you need to be. If you want to be thought of as something other than a propagandist, take what you write more seriously.