by David Safier
It's just too hard to resist. Easy to read stats and charts, easy to understand conclusions, a bit of punch to the results. The Goldwater Institute should be in marketing, not think-tanking. Oh, wait, that's right. It is.
Last Thursday I showed the methodological weaknesses of two G.I. studies that "conclude" private schools are better at building student character and get better marks from their students than public schools. Any statistical relevance of the studies' conclusions are wiped out by the poor sampling on the study.
I guessed that G.I. put out the studies as a counter to the very bad press tuition tax credits and STOs are getting. It needed to put a positive spin on private schools as well as tuition tax credits. And . . .
I ended with this statement.
Ladner can score one success with the Star. From this morning's Education Notes:
The conservative think tank, based in Phoenix, commissioned an outside company to survey public- and private-school students about the academic environment and racial and political tolerance of their high schools.
Then it lists four "findings" of the studies.
Rhonda Bodfield, the Star's education writer, is gone until September 15, so she doesn't get the hit on this one. Whoever put this in saw a ready-made, pre-packaged item and ran with it. Wanna bet the reporter didn't make a single call to anyone to check whether the study was valid or the conclusions make any sense?
Discover more from Blog for Arizona
Subscribe to get the latest posts sent to your email.
Matthew.
I appreciate that you agree with my point that your study lacks adequate statistical/methodological controls. However, your “everybody does it” rationalization doesn’t really work. When you compare Ds and Rs in a study, so long as you’re doing a good job of getting a random sample of both groups, your results have some validity. Same with men and women. But here, you need to say that you have two otherwise similar groups — students — who only vary in one way, the type of school they attend. I imagine each group studied was a reasonably random collection, but if the groups weren’t equivalent to one another, the results mean very little. (What if a political study compared predominantly low income Rs with predominantly high income Ds? The study would be laughed out of town. Your study merely raises a chuckle, but for the same reason.)
By contrast, two recent studies — public and private in one case, traditional public and charter in the other — have a very good possibility of being as valid as these studies ever are. Both in D.C. and in N.Y., the studies looked at two groups of students in a random lottery. One group was admitted to the private or charter school, while the other remained in a public school. The chance that these two groups are equivalent are pretty high, so I looked at the results with interest. With your groupings, not so much.
You keep trying to go back to discussing your results. I won’t bite. My point was, the results mean nothing because the methodology was poor. You appear to agree about the methodology. You should also agree that the results are only worth discussing as a matter of conjecture, not statistical analysis. The bar charts in your reports with precise percentages attached to them are just so much window dressing — eye candy to attract the story-hungry journalist. If you made a careful, dispassionate analysis of your own motivations in putting together the study, I’m confident you would agree with me, even if it might hurt your pride a bit.
David-
These results were merely a straightforward poll of public and private students. When you see polls comparing the attitudes of men and women, or Democrats and Republicans, how often do you see statistical controls to account for other differences in the populations?
I’m going to go with “never.” Never is a strong word though, so let’s go with almost never just in case.
Don’t get me wrong, in some circumstances I’d like to see that done. If I were trying to make the absolute claim that attending a private school makes a person more politically tolerant, it would be appropriate.
That is not, however, what I have done here. A straight poll of private and public school students is more than adequate to test the cartoonish theory of some of your fellow private school choice opponents that private schools serve as intolerance boot camps. The poll was sufficient to test that theory.
The poll shows that many public school students take a dim view of their schools on a variety of important factors. This is important regardless of the private results. The polls show a limited level of political tolerance among public school students. This is also important regardless of what the private poll demonstrates.
Matthew.
Though you want to shift the discussion to the conclusions of your “studies,” my point is, you can’t draw any valid conclusion from the data because the methodology is so poor. If that’s true, and I believe it is, then the conclusions aren’t even worth discussing except as a matter of conjecture.
Here is the crux of my argument from the post ( http://arizona.typepad.com/blog/2009/09/fools-gold-private-schools-better-if-you-use-shaky-data-to-arrive-at-questionable-conclusions.html ):
“If Ladner’s studies can’t show that the two groups of students surveyed are similar in every way but the one important variable — attending a private or a public school — his results are worthless.
“Ladner realizes this, so he tries his best to show he has two equivalent groups, and fails miserably.”
Reading your comment above, it seems you agree with me that the two groups may be different, because you don’t have adequate data about the students to make a valid comparison. You blame that on the fact that the students can’t tell you their family incomes. My point exactly. The data on your two groups is shaky, so there may be no real difference in the attitudes of public and private students if you had equivalent samples, or the differences may be slighter, or the public school students may actually be more tolerant if you control for socio economic status. I don’t know and you don’t know.
You just say it doesn’t matter that your data may not create equivalent groups of students. I would agree if you were writing a thought paper. But you present this as a valid study. It isn’t. That’s my complaint.
David-
Sorry to be so tardy to respond. I made the mistake to attend back to back conferences without a laptop.
I’ll beg your indulgence to only hold me responsible for what I actually write, not for what you feel when you read what I write.
If I recall your point about the surveys correctly, you essentially argued that uncontrolled differences in the population will have influenced the results. This is of course true.
This was a survey of high school students. I know that I didn’t have a clue about my family income when I was in high school, so I wouldn’t trust the results of such an item if it were asked (and it would result in a large number of ‘I don’t knows’ that would minimize the effectiveness of the item as a control in any case).
The theory I was testing however was not “does attending a private school make you more poltically tolerant?” Rather, the theory I was testing was that espoused by education reactionaries “attending private school will serve to indoctrinate students into dangereous intolerant ideologies.”
As it turns out, private school students are substantially more tolerant than public school students in Arizona. That may or may not be because they attend private schools, but it certainly calls the evidence-free theory of choice opponents into grave doubt.
Most of the results have nothing to do with family income. You either think your school gives you challenging work, or you don’t. You either think that your school treats students equally regardless of race or ethnicity, or you don’t.
The results also beg the questions- why do we see higher levels of intolerance in the public schools? Why do so few Arizona public school students report that their school treats people equally regardless of race or ethnicity? Why do only 39% of Arizona public school students report that their fellow students are focused on academics?
Matthew.
“That’s not spaghetti, that’s linguini!” (Jack Lemmon, The Odd Couple. Matthau’s reply: “Now it’s garbage.”)
First, I intentionally wrote the “Shorter Matthew Ladner” to capture what it felt like, not what it said. Figured I’d get all Republican on you. (“Obama wants to take my grandmother to Indonesia/Kenya where he was born and kill her!!!!!”) So I don’t stand corrected. I stand caught. (See how easy that kind of apology is?)
If you want to respond to something, I spent a great deal of time deconstructing your arguments saying your two studies are valid statistical samples. It’s all of a piece. Taking one part out of context won’t make much sense. If you want to show how I’ve been unfair, try and make the argument that an impartial authority who understands the use of surveys and statistical analysis would grant validity to your conclusions from your surveys, I would very much enjoy seeing that. Remember, we’re not trying to decide if your conclusions make sense, only if they are valid based on the quality of your study.
Todd-
Wow- I want my promise ring back!
I checked in to BfA today and find it full of GI bashing. Where to start?
First, David needs to take off his unhinged glasses and reread the daily email that he completely mischaracterized. I never said that public universities were bad, merely that they had self-evidently done little to slow down out of control costs at private universities. If you doubt it, look up the figures or call Harvard and ask them what their tuition runs these days.
Second, Schlomach was making reference to the state portion of the total K-12 funding, not the total funding.
Second, David’s critique of the survey results is replete with errors. Glaring among these was a fanciful summary of the empirical research on voucher programs. The federal government has sponsored 11 ultra-high quality random assignment studies of education interventions, and only 3 of them show statistically significant results. Guess which intervention has the largest positive impact by a wide margin? Read for yourself here from the PI of the evaluation:
http://educationnext.org/lost-opportunities/
David-
On your critique, you’ve thrown a great deal of spagetti on the wall. To demonstrate that I am a good sport, however, pick whatever main point that you believe is best supported, and I’ll post a response on JPGB.
I have lost my small remaining respect for any of them in the recent days. First there was the off-the-rails writing by Bolick on Obama’s speech to school children and then the two recent education related releases on education they you have shown to be egregiously intellectually dishonest. The rest of the GI people I have long thought of as little more than hacks, particularly Schlomach and Dranias, but Bolick and Ladner seemed a little better. I guess not.