It’s barely a year since political sages vowed to kick the habit of a lifetime, and do without opinion polls. In May 2015 there were more surveys, more exhaustively crunched, than ever before, and yet they proved so wrong that many declared “never again”.
The referendum, however, has triggered a serious relapse. The rhythm of the news cycle and the mood of the rival camps has – once again – been dictated by daily doses of data. Yes, if you want a justification for lapsing back into the habit of politics by numbers, you can probably come up with one. December’s Oldham West byelection, for example, which Labour handily won after endless reports from the ground predicting Corbyn would be crushed, suggested that journalistic shoe leather was just as fallible as questionnaires. The truth is that with so much at stake, we simply can’t resist. We may have discovered last year that we can’t live with the polls, but this year, we have discovered anew that we can’t live without them.
So what have they shown? Earlier on in the long campaign, the answer was “nothing clear”, because there were two parallel pictures, with surveys online suggesting a dead heat, while more traditional telephone polls stubbornly predicted a double-digit lead for remain. To suspicious minds this may have looked like a chastened industry covering its behind. Two different methods producing two different answers would enable the pollsters to treat their results like Groucho Marx’s principles, and say “if you don’t like my results, I have others”. Such cynicism, however, now looks unfair, because the two methods have since converged, with both coming to predict a relatively close race, in which – in the final straight – remain enjoys a slight advantage.
But what real confidence, if any, can we have that – after last year’s debacle – the surveys will be right this time around? Not much, unfortunately. The big flaw unveiled in the thorough post-election inquiry for the industry, by Prof Patrick Sturgis, has not been satisfactorily addressed. The root problem, he found, was not last-minute jitters in the ballot box or inadequate turnout filters, but rather a brute failure by the pollsters to interview the right people.
A couple of door-to-door surveys run by academics and published long after the event did get election 2015 right. The big difference was that these surveys picked out voters’ names at random, and then kept hammering on their doors until they answered. The other polls, whether online or phone, give up on the hard-to-reach, move on to other phone numbers and email addresses, and thus fail to achieve a genuine mix. In 2015 it transpired that Tories, for whatever reason, were that bit harder to rouse, creating the big polling miss.
The pollsters have come up with all sorts of tweaks to correct their Conservative shortfall – fiddling around with the precise age or class mix, or the strength of the assumptions made about how such demographics shape your likelihood to turn out. Some played around with the ordering of questions, others took a new interest in what newspapers their respondents read, and others again imposed stronger assumptions on which way “don’t knows” would go. None of it, however, touches the fundamental problem of failing to reach a properly politically representative mix of voters.
When I explained various post-election wheezes the UK industry has tried to the polling expert Nate Silver and his colleagues on the FiveThirtyEight podcast this week, their verdict was that it “sounded like you’re trying to rebuild the plane at the same time as trying to fly it”. Who is to say whether ad hoc fixes, to boost the Tories and knock Labour down by a few points, will improve things, or make them worse, in the very different context of a referendum that cuts across party lines?
The problem with online polls is that they rely on membership of self-selecting panels, which means they will – necessarily – over-represent the type of person who, in response to an email saying “Would you like to give your opinion on X or Y?”, answers “yes, yes, yes”. Listeners to talk radio have always known that the sorts who are bursting for the chance to vent their views are disproportionately opinionated and extreme, and – indeed – Ukip voters are systematically over-represented in raw internet data.
A different skew, which is getting deeper, is at work with the phone polls. In the world of the ubiquitous sales call, the sort of person who picks up a landline and answers the opener, “Hello, I’m from ABC research, I wondered if I could ask you a few questions?”, by saying “Sure, I’ve got time to talk” is not typical either. Response rates are plummeting, and those few who do pick up would – I suggest – be more inclined than the average to be sunnily disposed towards the rest of humanity. Labour was strongly over-represented in several of last year’s telephone polls.
Now one might, very tentatively, venture that my suggested online bias – towards the hyper-opinionated – would work to flatter leave. ICM analysis, which pointed to a particularly high proportion of leave-leaning voters rushing to reply to online survey requests within the first hour of them being sent out, gives some support to this. Conversely, my hunch would be that my suggested telephone bias – towards those who feel warmth towards strangers – would flatter remain. But who knows? After all the jiggery-pokery, whether by accident or design, the two survey modes have converged.
So what evidence do we have that is truly independent of the basic phone and internet data? One mostly online survey, conducted by academic researchers at NatCen, was distinguished by its attempt to achieve something closer to a real random mix, by recontacting people who didn’t immediately reply. It pointed to a 53%-47% victory for remain, so a shade more decisive than the average 52%-48% lead for remain in the final polls. The trouble with this is that the fieldwork took place between 16 May and 12 June, so most of it missed the early June swing to leave reported in all other polls, never mind the final week swing back to remain. Slightly more timely is a mid-June BMG phone poll, which also made a proportion of repeat calls, and separated out the results. This suggested that, like 2015 Tories, remain supporters were more likely to require a second call, somewhat harder to get hold of, and thus – perhaps – likely to be under-counted.
One last-gasp rationale for keeping faith with the polls is that although they are all flawed, at least they are flawed in different ways. That gives some reason to have faith in those things they all have said consistently. Like, for example, the fact that the elderly are more leave-inclined, and graduates more likely to lean to remain. Like, too, the swing towards leave in mid-June, and the swing back in the past few days.
Most, but not all of them, now rank remain as enjoying the edge. Put that together with the BMG finding about remainers being more time-consuming to call, and you begin to see why the betting markets are so convinced that Britain is staying in. So the head says the final polls will probably have it right, and remain will indeed have the edge. But – for my own part – I’m certainly not rushing down to the bookies. The one racing certainty is that if the pollsters get it wrong again tonight, they’ll be out in the cold for more than a year this time.