As the media narrative turns: Democrats on path to retain the Senate

The media villagers love polling because it allows them to fill column space and empty hours of air time with speculation and conjecture. It’s easier than doing actual reporting about candidates and issues. Unfortunately, this is a media culture that is not going to change.

For much of this year the media villagers have been relying on “fundamentals” polling — comparing what has occurred in past elections to predict future election results  — to support their preconceived media narrative that Tea-Publicans will take control of the Senate in this midterm election because the “fundamentals” say so.

Do the “fundamentals” have some value in interpreting polls? Yes. But elections are like snowflakes, each is unique, with a different electorate, candidates, and issues that drive the results in each election. No two are alike. As I have said before, “One of these things is not like the other.” It is a fundamental error to assume that correlation is causation. See some hilarious charts showing that correlation is not causation.

More reliable projections come from poll-focused analysis. Sam Wang at Princeton University, who correctly predicted every senate race in 2012 using his model, is much more bullish on Democrats retaining the Senate. Senate 2014 Election Day Model (geeky version):

Those of you who followed PEC in 2012 will recognize all the components. Basically, I have adapted the Presidential model for the Senate. Here I will document key components that are needed for the Election Day prediction.

First, let me summarize the core principles of any model at the Princeton Election Consortium.

1. Polls are the only inputs, and are never “corrected.” Median-based statistics are used to reduce the influence of outliers.
2. Fundamentals based in political science have a minimal role.
3. The most important output of the model is the Meta-Margin (and not the win probability).
4. Polls from earlier in the year are used to predict future outcomes.
5. PEC’s results let you direct your attention and activism to the races that matter most.
6. All the code is open-source.

Today I will focus on principles 2, 3, and 4. They involve assumptions that should be examined critically.

Principles 2 and 4 are the biggest differences between us and NYT/FiveThirtyEight, which use models that are based on polls, but also draw upon non-polling fundamentals to nudge expectations for where the race ought to be. The general gist of the fundamentals is that political conditions this year favor the Republicans. Since Democrats are currently outperforming those expectations, the other models predict that in the next two months, Senate polls will drift toward the Republicans. At the core, this is why those models lean more than we do toward the possibility of Republican control of the Senate in 2015.

However, this year a fundamentals-based correction is fraught with difficulty. The reason is that the Senate is likely to split fairly close to 50-50 between Democrats+Independents and Republicans. In this circumstance, even a small amount of imprecision in the non-polling-based estimates can send a prediction awry. In fact, I estimate that FiveThirtyEight’s consideration of fundamentals has the effect of shifting the polls toward Republicans by about 2.0 percentage points. That may not seem like much, but it is enough to flip the sign of the prediction.

To illustrate the difficulties associated with a fundamentals-based prediction, let’s take the House of Representatives as an example. In midterm elections, on average, the Congressional ballot preference moves away from the President’s party from Labor Day to Election Day. However, in the last four midterm elections (2010, 2006, 2002, and 1998), the ballot preference has gone toward the President’s party, away from it, or not changed at all. That is a demonstration of the fact that fundamentals-based models are quite variable in their predictions, perhaps too much so to help.

This year, the data so far indicate that fundamentals aren’t matching the polling data so far. On average, in midterm elections, opinion usually goes against the President’s party. But here is what the generic Congressional ballot looks like.

Most of this graph is not very far from the blue line, which labels the national House vote in the 2012 election. In that respect, the generic Congressional ballot is looking rather different from the campaign of 2010, when opinion favored Republicans all year. In other words, the average fundamentals-based expectation for the House is not being met in 2014.

Now let’s look at Senate polls. Here our best source is PEC’s own Senate Meta-Margin, an extremely valuable tool. The Senate Meta-Margin is defined by how much polls would have to be shifted, across all states, to bring the Democratic/Independent control probability to exactly even odds for the two parties. Since the Meta-Margin is in units of public opinion, it’s a convenient quantity to think about, and to model.

Here is this year’s graph for the Senate Meta-Margin.

The most apparent quality of the Meta-Margin is that it has been above the red line, i.e. favoring Democrats+Independents, for most of this year. (The last big jump at the start of September reflects the withdrawal of Chad Taylor (D) in the Kansas Senate race, which opened the way for independent candidate Greg Orman.) Also, it isn’t moving strongly in either direction. To put it simply: Democratic candidates have steadily outperformed the fundamentals-based models.

Wang gets really wonky at this point, but his conclusion is: “In other words, without equations: the probability that Democrats and Independents will control 50 or more seats is 70%. Because of uncertainties, this probability has a likely range of 50 to 80%.”

The Huffington Post has entered the polling game this year and is also doing poll-focused analysis like Sam Wang.  HuffPollster:

HuffPost Pollster begins by collecting every publicly released poll on the 2014 Senate races. We then use a statistical model to estimate the trend in support for each candidate based on all the survey data, adjusting for sample size and pollsters’ “house effects.” Interactive charts of those support trends are available here and on the HuffPost Pollster home page.

By running a series of simulations (known commonly as the Monte Carlo method), the model allows us to quantify the uncertainty associated with the current polling snapshot. That uncertainty comes from multiple sources: sampling error in the polls themselves, uncertainty about the house effect corrections, and uncertainty about how quickly vote intentions are changing.

The model then calculates a “win probability” for each race that is displayed in the graphics on this page. This probability takes three factors into account:

• The time remaining between the current snapshot and the election.
• The possibility that the polls could be wrong or that some sort of major event could shake up a race in ways that the current polls can’t measure.
• The proportion of “undecided” voters in the polls. If the undecided proportion is high relative to the expected margin between the candidates, the outcome of that race must be less certain.

 Lastly, we combine the win probabilities from each race and perform another set of Monte Carlo simulations to calculate the likelihood of each outcome — giving us the probabilities of Democrats keeping control of the Senate or of Republicans taking over.

HuffPollster also projects the Democrats retaining the Senate.

Even the “fundamentals” polling analysis has begun moving closer in line with the poll-focused analysis. The Washington Post’s Election Lab has Dems very narrowly favored to win the Senate for the first time this year. Ben Highton writes at the Posts’ Monkey Cage blog today, Good news for Senate Democrats. Maybe:

There are two central reasons why we estimate better chances for the Democrats now than before.

First, as Election Day approaches, our model gives increasing weight to polls (especially recent polls) rather than background fundamentals, such as incumbency and state partisanship.  As we have discussed, early in the campaign season, taking into account fundamentals helps with forecasts.  Later in the season, the polls are pretty much all anyone needs.  (See also what Josh Katz at The Upshot has to say about polls and fundamentals.)

Second, there is a disproportionate number of elections where Democrats are polling better than one would expect based on the fundamentals alone, a phenomenon also discussed by Harry Enten and Sam Wang.  In fact, if we compare the estimated probability of winning based only on fundamentals to the current estimated probability of winning that includes the polls, Democrats do better in six of the seven elections where the differences are at least 10 percent.

What explains this over-performance by Democrats, or under-performance by Republicans?  One possibility is that the “midterm penalty” — the loss in vote share suffered by the president’s party in the midterm — is shaping up to be smaller than in the past  . . .  [I]t is plausible that the size of the midterm penalty in 2014 may end up being smaller than in the past.  This could be the consequence of voter discontent with the Republican Party, as Nate Cohn has noted.

Another possibility is that there are idiosyncratic features of individual races that the background fundamentals cannot easily capture, and which favor Democrats in certain races. For example, maybe some candidates in the key races are just better or worse in ways that we cannot easily measure — but that the polls are capturing.

Even Nate Cohn at the New York Times‘ The Upshot, who has been a real “Debbie Downer” all year, is starting to come around slowly. Democrats Are Seeing More Daylight in Path to Senate Control:

A few months ago, the Democratic path to a Senate majority looked long and arduous. It has started to look easier.

* * *

But today the Democratic path to victory looks as clear as it has at any point this year. That path remains narrow, to be sure. The Democrats will probably still need to sweep those five fairly close races. Yet with just two months to go, the party appears to have an advantage in four of them. And the Democrats have other opportunities that might give them more breathing room.

* * *

Over all, the Republicans are still the slightest favorites to retake the chamber. For Democrats to retain their majority, they will have to win at least two states that voted against President Obama in 2008 and 2012. It is possible that this will prove untenable over the final few months of the race, and that Republicans will gradually gain as undecided voters who disapprove of the president’s performance and voted for Mitt Romney make up their minds. But the Democrats now have a lead in enough races to get to 49 seats — and have a few options to reach 50.

As always, there is only one poll that counts: Election Day, and that comes down to who can turn out their base voters to the polls.