Rating the media villagers and Beltway bloviators

Posted by AzBlueMeanie:

A Hamilton College class and their public policy professor analyzed the predictions of 26 pundits — including the Sunday morning bobbleheads — and used a scale of 1 to 5 to rate their accuracy.

The data is dated and limited. The period of time the study examined is from September 2007 to December 2008, essentially the 2008 election cycle.

While the study is somewhat interesting, there is one glaring omission: "If it's Sunday, it's John McCain." Nobody, but nobody, appears on the Sunday morning bobblehead shows more than John McCain and nobody, but nobody, has been demonstrated to be as consistently wrong in his predictions as John McCain. I have to question the results of any study that did not include John McCain (his BFFs Joe Lieberman and Lindsey Graham make the "ugly" list, which indicates where he would rank).

Here is the executive summary explaining how the results were obtained. Hamilton College – Pundit – Executive Summary:

In this paper, we report on the first-ever test of the accuracy of figures who made political predictions. We sampled the predictions of 26 individuals who wrote columns in major newspapers and/or appeared on the three major Sunday television news shows (Face the Nation, Meet the Press, and This Week) over a 16 month period from September 2007 to December 2008. Collectively, we called these pundits and politicians “prognosticators.” We evaluated each of the 472 predictions we recorded, testing it for its accuracy. Based on an analysis of these predictions, we answer three questions:

1. Which prognosticators are most accurate? We found wide disparities in the predictive accuracy of these individuals, and we divided them into “the good, the bad, and the ugly.”
    
2. Which characteristics are associated with predictive accuracy? We examined the effects of age, education, ideology, and other factors on accuracy.
    
3. What is the purpose of media pundits? We discuss whether the ordinary citizen should look to pundits for deeper analysis of events, or whether pundits are simply a more enjoyable way to learn about the events of the day. We also consider alternative viewpoints, including the notion that pundits are useful as representatives of opposing points of view in the country, and the idea that they are simply entertainers.

Continue reading the Hamilton College – Pundit – Executive Summary.

Here is the full report. http://www.hamilton.edu/documents/An-Analysis-of-the-Accuracy-of-Forecasts-in-the-Political-Media.pdf.

The Good

The Good comprised the best of our prognosticators, each of them with a PVS over 5.

1. Paul Krugman, New York Times. (Best Rated)
2. Maureen Dowd, New York Times.
3. Ed Rendell, Governor of Pennsylvania.
4. Sen. Chuck Schumer (NY).
5. Kathleen Parker, Washington Post (Caveat: Kathleen Parker made only six testable predictions, barely qualifying her to be included in our regression. The majority of her five predictions focused on what was happening in Republican politics in the lead up to the 2008 election).
6. Rep. Nancy Pelosi (CA), Speaker of the House.
7. David Brooks, New York Times.
8. Eugene Robinson, Washington Post.
9. Hank Paulson, Secretary of the Treasury.

The Bad

“The Bad” is comprised of those pundits whose Prognosticator Value Score is at or above a 0 but below a 5. In other words, they are more reliable than a coin toss, but less reliable than those who are considered “The Good.”

1. Newt Gingrich, former Speaker of the House. (Caveat: All but a few of his predictions dealt directly with the primary and presidential elections).
2. Mike Huckabee, former Governor of Arkansas and presidential candidate.
3. Howard Wolfson, Communications Director for Hillary Clinton’s 2008 presidential campaign.
4. Sen. John Kerry (MA).
5. Bob Herbert, New York Times.
6. Andrea Mitchell, NBC News/MSNBC reporter.
7. Thomas Friedman, New York Times.
8. David Broder, Washington Post.
9. Clarence Page, Chicago Tribune.
10. Nicholas Kristof, New York Times.
11. George Will, Washington Post.
12. Sen. Hillary Clinton (NY) and presidential candidate.

The Ugly

Five prognosticators had negative PVS, meaning each person in the following group was wrong more often than they were right.

1. Sam Donaldson, ABC News reporter.
2. Sen. Carl Levin (MI).
3. Sen. Joe Lieberman (CT).
4. Sen. Lindsey Graham (SC).
5. Cal Thomas, Chicago Tribune. (Worst Rated)