Nate Silver confirms our opinion – Rasmussen Reports suck!

Posted by AzBlueMeanie:

Every election cycle has its winners and losers: not just the among the candidates, but also the pollsters. Numbers guru Nate Silver confirms our opinion that Rasmussen Reports suck! Rasmussen Polls Were Biased and Inaccurate; Quinnipiac, SurveyUSA Performed Strongly:

On Tuesday, polls conducted by the firm Rasmussen Reports — which released more than 100 surveys in the final three weeks of the campaign, including some commissioned under a subsidiary on behalf of Fox News — badly missed the margin in many states, and also exhibited a considerable bias toward Republican candidates.

Other polling firms, like SurveyUSA and Quinnipiac University, produced more reliable results in Senate and gubernatorial races. A firm that conducts surveys by Internet, YouGov, also performed relatively well.

* * *

The 105 polls released in Senate and gubernatorial races by Rasmussen Reports and its subsidiary, Pulse Opinion Research, missed the final margin between the candidates by 5.8 points, a considerably higher figure than that achieved by most other pollsters. Some 13 of its polls missed by 10 or more points, including one in the Hawaii Senate race that missed the final margin between the candidates by 40 points, the largest error ever recorded in a general election in FiveThirtyEight’s database, which includes all polls conducted since 1998.

Moreover, Rasmussen’s polls were quite biased, overestimating the standing of the Republican candidate by almost 4 points on average. In just 12 cases, Rasmussen’s polls overestimated the margin for the Democrat by 3 or more points. But it did so for the Republican candidate in 55 cases — that is, in more than half of the polls that it issued.

If one focused solely on the final poll issued by Rasmussen Reports or Pulse Opinion Research in each state — rather than including all polls within the three-week interval — it would not have made much difference. Their average error would be 5.7 points rather than 5.8, and their average bias 3.8 points rather than 3.9.

* * *

Rasmussen’s polls have come under heavy criticism throughout this election cycle, including from FiveThirtyEight. We have critiqued the firm for its cavalier attitude toward polling convention. Rasmussen, for instance, generally conducts all of its interviews during a single, 4-hour window; speaks with the first person it reaches on the phone rather than using a random selection process; does not call cellphones; does not call back respondents whom it misses initially; and uses a computer script rather than live interviewers to conduct its surveys. These are cost-saving measures which contribute to very low response rates and may lead to biased samples.

Rasmussen also weights their surveys based on preordained assumptions about the party identification of voters in each state, a relatively unusual practice that many polling firms consider dubious since party identification (unlike characteristics like age and gender) is often quite fluid.

* * *

Rasmussen Reports has rarely provided substantive responses to criticisms about its methodology. At one point, Scott Rasmussen, president of the company, suggested that the differences it showed were due to its use of a likely voter model. A FiveThirtyEight analysis, however, revealed that its bias was at least as strong in polls conducted among all adults, before any model of voting likelihood had been applied.

Some of the criticisms have focused on the fact that Mr. Rasmussen is himself a conservative — the same direction in which his polls have generally leaned — although he identifies as an independent rather than Republican. In our view, that is somewhat beside the point. What matters, rather, is that the methodological shortcuts that the firm takes may now be causing it to pay a price in terms of the reliability of its polling.

* * *

The table below presents results for the eight companies in FiveThirtyEight’s database that released at least 10 polls of gubernatorial and Senate contests into the public domain in the final three weeks of the campaign, and which were active in at least two states.

The most accurate surveys were those issued by Quinnipiac University, which missed the final margin between the candidates by 3.3 points, and which showed little overall bias.

The next-best result was from SurveyUSA, which is among the highest-rated firms in FiveThirtyEight’s pollster rankings: it missed the margin between the candidates by 3.5 points, on average.

* * *

Other polling firms that joined Rasmussen toward the bottom of the chart were Marist College, whose polls also had a notable Republican bias, and CNN/Opinion Research, whose polls missed by almost 5 points on average. Their scores are less statistically meaningful than that for Rasmussen Reports, however, because they had only released surveys in 14 and 17 races, respectively, as compared to Rasmussen’s 105 polls.

What Nate Silver's analysis cannot answer is how much effect the prevalence of GOP narrative pollster Rasmussen Reports being reported by the mainstream media as a credible poll swayed public opinion toward Republican candidates. A persuasion effect of less than two percent would have been the difference in a number of races. Polls can effect voters' perceptions of a race.

And that is why there is a cottage industry for producing GOP narrative polls. It would be better if the news media simply did not report on polling data at all, just let the voters decide. But then what would the media villagers and Beltway bloviators have to speculate about 24/7 — instead of, oh I don't know, actually reporting on real news stories that are relevant to the lives of people? Nah, that would be work, and we can't have that.