More on the G.I. Civics test

by David Safier
My post on the Goldwater Institute's Civics test which showed high school students ignorant about basic U.S. civics questions has created quite a hubbub in the comments section, including a few comments from the study's author and friend of the blog Matthew Ladner. Rather than adding to the string of comments, I'll put my thoughts here.

When I questioned the accuracy of the survey results, Ladner protested in his commments that a phone survey is a valid instrument and it was conducted by a reputable survey firm. He also said I was wrong to say the survey was a hit piece on public schools, since it included results from private and charter schools as well.

As commenter Flounder Todd mentioned and I'll repeat, the email Ladner wrote that goes out to the G.I. mailing list only listed the public school score on the Civics test: 3.5 percent. You can find the entire study on the G.I. website, but let's face it, most people will stop at the email. Leaving any mention of the charter and private school results out of the email indicates to me that public schools were Ladner's prime target.

Ladner doesn't address my comment about it being intellectually dishonest — not incorrect, but dishonest — to use a precise percentage like 3.5 percent to represent the results of a very imprecise, non-random phone survey, even though it was one of my primary criticisms. An intellectually honest representation of the findings would have said that a non-scientific phone survey showed that under 5 percent of public high school students passed. That's almost equally scathing, but the number-to-one-percentage-point implies a level of scientific accuracy that gives more weight to the finding than it deserves. Any good statistician will tell you not to imply a degree of certainty for your data that doesn't exist.

As for a phone survey being a good way to get accurate information . . . I'll accept that, with qualifications, for adults. But for teenagers? Uh uh. If you're asking about music preferences, maybe. If it's a quiz about video games, maybe. If it's about civics and the answers are multiple choice, maybe. But just asking a question like, "What is the supreme law of the land?" and expecting teens sitting at home to ponder the question and give the best answer they can come up with . . . uh uh, it ain't gonna happen with most of them. If you think they'll give the questions their full attention, you haven't hung around with a random assembly of teenagers lately. If they don't have the answer right on the tips of their tongues, they're likely to say either the first thing that comes to mind or "Don't know."

(I would like to see the script the survey firm used to introduce the questions. If it didn't represent the nature of the questions accurately, lots of the teens surveyed would have said, "What the . . ." after the first few questions and gotten through the rest of the survey as fast as they could.)

One more point, something I haven't mentioned before. Before potential new citizens go in to take the test, they're given a full list of 100 questions to study, right? And told they'll have to answer 10 of them? If I'm right, and I'm pretty sure I am, that means they don't come in cold. This is much easier than studying for a driving test. It's pure rote memorization, and the test takers are more than willing to put in the time, because the reward at the end of the test is U.S. citizenship. No wonder 92.4 percent pass! How many of them — or how many adult citizens — would do well on the test if they took it cold, like the students did? To make the comparison Ladner does, between people eager to become citizens who studied the hundred questions and teenagers getting the questions cold over the phone, is another example of intellectual dishonesty. It's a false comparison designed to make the students' lack of civics knowledge look as bad as possible. That kind of thing is fine for Jay Leno. He's an entertainer. G.I. is supposed to be a serious, reputable institution.

Again, I'm not defending the students' lack of knowledge. Their ignorance is not surprising, but it's not good either. I'm saying Ladner's piece is an example of "shock and awe" reporting, where making your point is more important than representing the findings accurately. True, that kind of thing happens on the left and the right, but for the Goldwater folks, it's their basic M.O.