General election opinion poll failure down to not reaching Tory voters

The long-awaited postmortem into what wrong with the opinion polls ahead of last year’s general election is published on Tuesday, and it points the finger at pollsters’ failure to reach enough Conservative voters.

Patrick Sturgis, a professor of research methodology at Southampton University, who headed a team of nine experts that undertook an independent review for the British Polling Council, said “the emerging upshot is that the companies are going to have to be more imaginative and proactive in making contact with – and giving additional weight to – those sorts of respondents that they failed to reach in adequate numbers in 2015.”

Emerging findings across the industry point to a series of key groups who were underrepresented in the polling:

• The oldest voters: the over-70s, who broke heavily for the Tories, were not reflected in YouGov’s online internet panels.
  
• Young non-voters: the under-30s generally lean left, but very often fail to turn out on polling day. The pollsters, however, reached an atypical group of youngsters, who were unusually engaged with politics and committed to voting.
  
• Busy voters: in the face-to-face British Social Attitudes survey, Labour was six points ahead among respondents who answered the door at the first visit, whereas the Tories enjoyed an 11-point advantage among interviewees that required between three and six home visits. Even after adjusting for social class and age, those easy-to-reach voters are less Conservative than the “busy” respondents the pollsters have to work hard to chase.
  

Sturgis’s conclusions were informed by three recent analyses of YouGov, British Election Study and British Social Attitudes data. He explained that there was “no single, straightforward fix for not having the right sort of people in your sample”.

Sturgis also downplays suggestions of a material, last-minute swing to the Conservatives, concluding that if it happened at all “it only accounted for a small amount of the total polling error”.

He risks the displeasure of the industry that commissioned him by highlighting one “surprising feature of the 2015 election”, namely “the lack of variability across the polls”. The surveys were, he finds, more closely bunched around the prediction of a Conservative-Labour dead heat than should have been expected, because of random variation alone.

Sturgis is at pains to emphasise he has found no hint of malpractice, but the pollsters’ critics may charge them with using the weighting schemes and other adjustments they apply to their samples to produce their final estimates in order to protect themselves from breaking away from the pack.

Sturgis’s final report, which will include detailed recommendations on what pollsters need to do differently, is not due out until March, but he is already clear that “there needs to be a shift in emphasis away from quantity and towards quality”.The latest regular Guardian/ICM telephone poll, released on Monday, puts the Conservatives on 40%, Labour on 35%, Ukip on 10% and the Liberal Democrats on 6%.

In the light of the general election debacle, ICM has already adjusted its methods to toughen up the assumptions it makes about the votes of people who decline to reveal their intention. Without such changes, its director Martin Boon explains the Tories would be on 39% and Labour on 37%. He cautions that there are still signs that this may be overstating Labour strength. “Thirty-five percent is probably too high. We can see in the small print of this poll that we’ve still got too many respondents who recall voting Labour,” he said.

The Sturgis review conducted a detailed review of 27 polls from nine British Polling Council members across the short election campaign last year. The data covered 47,196 individual respondents. It is a mark of how close a result the polls predicted that of these respondents, 15,291 said they intended to vote Conservative, statistically indistinguishable from the 15,368 who said they would vote Labour.

Speaking on BBC Radio 4’s Today programme on Tuesday, Sturgis said that the failure to predict May’s result did not signal the death of political polling.

“Although the result was wrong, and that is always the worst thing to get wrong, actually the polls were pretty bang on in terms of the smaller parties. They got the Greens, they got the SNP surge, they got the collapse of the Lib Dems. Those were all exactly right,” he said.

“You could look at it in some ways as a bit unfortunate that they were three points over on Labour and three points under on the Conservatives, but the fact is that the polls are by far and away the best way of trying to figure out what the election result is going to be.”

Sturgis also said it was time for people to recognise that pollsters were not infallible. “Even if we move to the most expensive random survey that you can possibly imagine, there would still be a chance that you would get it wrong,” he said.

Ben Page, the chief executive of Ipsos Mori, told the Today programme: “If you look at the opinion polls at all of the elections since 1959, on average the polls have overstated the Labour share of the vote by 1.4%. The difficulty is [that it hasn’t happened] at every election. Our 2010 results were pretty good, 2005 too. What we saw this time was a disproportionate over-claim of turnout, particularly by younger Labour supporters.”

Page said that polling done with modest budgets and under huge time pressure would always risk being inaccurate: “There are some really interesting questions about whether we should stop doing [political polling] altogether, or we should say – as with the exit poll, which was far more accurate but cost around £200,000 – that we’re not going to do it unless the media sponsors are willing to invest the money seriously.”