And the winner is... Face validity of the polls 2011.

"Research shows that 90% of research is wrong." – Bob Newhart

Polling is one of the rawest forms of market research. Unlike market research for business which is not cost-effective to properly validate (managers just want data that is “good enough”), polling numbers get validated on election day. The trouble is there is so much excitement about what happened once the election results are tallied up, that few folks circle back to check out which polling firm provided the best prediction for election day. Today, I am going to do exactly that. I am going to revisit four of the big market research firms (Ipsos, Nanos, Leger, Ekos) who published polls immediately before the Federal Canadian election. Which firms got it right? Which firms got it wrong? Are there any polling biases? How good is the poll-of-polls?

The federal election polls that I am looking at are these ones:

Apr-29 IPSOS sample size: 1,710
May-01 NANOS sample size: 1,068
Apr-30 LEGER sample size: 3,521
May-01 EKOS sample size: 2,988

All look at national public support for each political party brand. By the way, Leger and Ekos have the largest sample sizes (3521 and 2988) and, theoretically (all things being equal), they should have the most accurate responses. We'll see below this is not the case.

Assumption

In the analysis that follows, I am making the assumption that voters did not change their political party brand voting preferences between date of data collection (which is done prior to the date of poll publication) and election time. This is not a trivial assumption to make in this year’s election because on May 1st, a major geo-political story broke (the killing of Bin Laden) that could have affected some voters. Nevertheless, this poll-validating is an opportunity we get once every few years- so we better capitalize on it.

How accurate are the pollsters? The poll-of-polls?

The first thing I'll check out here is the face validity of results. Who got the rank order of popular support right? Overall, Ipsos, Nanos, Leger, and the poll-of-polls (Average) predict the popular support rank order of all political brands accurately. Ekos got the major political party ranks right, but switched the ranks of the Green and the Bloc.



Let's get a little bit more serious now and look at some numbers. In this check, I summed up the discrepancy between the % votes that each political party received on election day and the final poll numbers that were assigned to each political party by each polling firm. Here we see a clear “winner” and “loser”. Ipsos only had a 5.3 point total discrepancy. Ekos had a 10.3% discrepancy. That’s quite a difference. Interestingly enough, the poll-of-polls is not the most accurate, although it is pretty close to being so.



Starting to scratch a little deeper now, we can look at how each polling company fared at the individual political party level. The graphs speak for themselves. “Actual” is the actual % of votes on election day.









Again, the Ipsos poll tends to be the most accurate. It was dead-on for predicting the Liberal brand vote and the Green brand vote. Ipsos was also the most accurate on predicting the Conservative brand vote. That being said, Ipsos blew it on the NDP brand support. (Check out the Ipsos NDP inflation. That result appears to be well outside the margin of error.) Nanos appears to be the next most accurate poller. While Nanos underplays the Conservative support, it’s still a lot less "less accurate" than Ipsos is when it is off on NDP support.

Polling Bias or not showing up at the polls?

Here's an interesting observation. The Conservatives brand support was the only brand that was consistently under-reported in all of the published polls. In contrast, NDP support was (generally) over-reported in the published polls. This suggests a few things. Most obviously, there may be sampling bias in the polling. For whatever reason, Conservatives may just be tougher to reach and NDP supporters more accessible. Second, it could also mean that NDP (and to a lesser extent Liberal, Bloc and Green) supporters were a lot less committed to voting. Prevailing wisdom, however, is that when citizens want change, they are more committed to voting.

Is the the Poll of Polls the King of Kings?
The poll-of-polls keeps getting more and more share-of-airtime. So how good is it? Looking at the graphs above, only once (the Bloc graph)does the poll-of-polls provide the most accurate reflection of political party support (compare this to Ipsos which is 3/5). That being said, by smoothing outlier results (e.g. Ipsos NDP, Ekos on Conservatives), we see that the poll-of-polls gives reasonable bounds for each political party. In hindsight of this election, the poll-of-polls provides a useful tool to see what is “not likely to happen”. (IE it caught the Ekos error on Green brand support exceeding the Bloc's).