How the 2012 election polling really was skewed for Mitt Romney

Harry Enten, guardian.co.uk

There is a fear runs through any good prognosticator's bones when dealing with a seemingly close election. Polls are not perfect. They are instruments to judge public opinion, and they can be wrong.

2012 also had the added feature of Republican confidence. You didn't have to look very far to hear the word "skewed" in response to polling data. The polls had too many Democrats, or so the claim went. Republicans were also banking on national polls that were kinder to Romney than state surveys.

The national polls, as it turned out, were not systematically biased against Republican Mitt Romney. The final surveys pointed to President Obama's re-election, and they were right. That does not mean, however, that the polls weren't biased. It won't be until all the ballots are counted that we can access who the most accurate pollster was on the state level, but we do know that Mason-Dixon and Rasmussen were off the mark. Right now, we can only make statements about national polling.

We know that the national surveys tilted heavily against Obama. When we don't count any one survey date twice (that is, tracking polls such as from Gallup only have each day counted once), we can say that the average of national polls taken after the first debate through election day had Obama winning by 0.3 percentage points. President Obama currently has a 3.2pt lead nationally and it seems like he may finish with an edge above 3.5pt.

Let's say you don't want to count Gallup and Rasmussen because you judge that they were biased. The average of non-Gallup and non-Rasmussen surveys after the first debate had Obama leading by 0.7pt: that's still going to be an error of at least 2.5pt and perhaps nearly 3pt.

You might think it's unfair to count polls taken so far back from election day (although most scoring techniques utilize a similar method to the one employed here). Even if you only count the surveys done over the final week of the campaign, Obama's lead is 1.3pt; that'll be an error of around 2pt. Eliminating Gallup and Rasmussen put the final average up at 1.5pt, yet, that is also going to be perhaps up to 2pt biased against Obama.

Some math indicates that internet polls were more accurate than other surveys. That's true to an extent, yet internet polls were biased, as well. Over the final month, an average of internet polls gave Obama a 1.6pt lead, to give an overall bias of at least 1.5pt, and perhaps near 2pt. For example, an average of all Google polls, post-first debate, gave Obama a 1.4pt advantage.

Unlike the overall aggregate, internet polls were not too much more accurate in the final week than in the month before the election. They had Obama ahead by only 1.8pt. Counting each pollster only once (as Google had many one-day surveys in the final week), Obama is ahead by a slightly larger 2.2pt – at least a percentage point off the final margin.

What about the pollsters who did everything right, according to polling gold standards? Those who conducted surveys over the phone with people doing the interviewing and including cellphone calling? The average of those polls after the first debate had Obama ahead by an average by just 0.1pt. Eliminating Gallup puts the average at a slightly higher 0.5pt – in other words, way off the mark.

These live-interview cellphone calling pollsters were slightly better in the final week. They pegged Obama's lead at 1.2pt – still at least 2pt too low. Without Gallup, Obama's edge climbs to 1.5pt, which is not a whole lot better.

It's fairly clear that no matter what method is utilized, the national polls were too favorable towards Romney. Only eight out of 113 polls (or 7.1%) during the final month had Obama's lead at above 3pt. Three of the eight were from the Rand Corporation, two were from Google, and the rest were scattered between Democracy Corps, IBD/TIPP, and the United Technologies/National Journal survey.

During the final week, only three out of 30 polls (or 10%) conducted had Obama's lead above 3pt. They belonged to Democracy Corps, Google, and Rand. If Obama's lead climbs to 3.5pt, then, even with rounding, the vast majority of national polls were off in the final week. (Note: usually one would consider a poll giving Obama's victory margin as 3pt to be right if the final result were between 2.50 and 3.49pt.)

Only two pollsters with at least two surveys post-first debate out of about 30 (less than 10%) had Obama's average lead at 3.5pt or above. Again, it was Democracy Corps (Obama +3.5) and Rand (Obama +4.0).

The fact that it's these two pollsters that may come closest to the final result is interesting to say the least. Democracy Corps is a fine polling organization, though some might assume bias in an outfit founded by Democratic operatives James Carville and Stan Greenberg. Rand was a panel-back survey (that is, using the same respondents over and over again), which weighted respondentsaccording to how sure they were to vote and for whom. That is, a respondent might say they were only 60% likely to vote and 60% likely to vote for a particular candidate. There were doubts whether Rand's methodology was sound, but it clearly worked this year.

Overall, it's fairly clear that the national polls missed the mark, big time. Depending on the final count, it seems possible that less than 10% of surveys came close to correctly projecting Obama's edge. Few will talk about the national miss because Obama won. But just imagine that the same magnitude of error had resulted in almost all the polls showing a narrow Romney lead going into election day, but Obama had won by just 1pt. Then we'd be having a conversation.