Strong and stable? Theresa May's election poll lead succumbs to outlier syndrome

I hate to say I told you so but, tell a lie, I told you so. As the election campaign heads into the home straight, could the unthinkable be on the cards? Could Theresa May, who started some four weeks ago with what was widely regarded as an unassailable lead, be on the verge of losing?

That's not the same as Labour winning – the political earthquake required for that is still nowhere seriously contemplated. But if she can fail to achieve a landslide, who knows?

It's all in the numbers, or rather the polls, and the issue for anyone trying to interpret them right now is the presence of some downright contrary numbers, otherwise known as outliers.

What are they telling us? How can we make sense of the stream of often conflicting data that we are presently receiving from news media?

With Conservatives taking 47 per cent of the vote, as polls were suggesting at the start of the campaign – some put the share even higher – they will win. The only question is the size of their majority. That remains true, even with Labour clawing its way back up to over 40 per cent and a wipeout for every other English party.

Because the problem for Labour, and in a slightly different way, the Lib Dems, is the distorting nature of our electoral system: First Past the Post amplifies the benefit of increasing vote share to parties whose vote is evenly distributed, simultaneously punishing those whose vote share is more geographically targeted.

Add, too, the quirk that the growth of one minority party can inadvertently benefit another. UKIP taking Tory votes helps Labour; Lib Dems are helped by a higher Labour one.

That is why Labour can withstand being squeezed down to 20 per cent and still hold around 100 seats while the peculiar circumstance where three parties achieved parity in the national vote share, each getting exactly 30 per cent, would give the Tories 271 seats, Labour 242 and Lib Dems 60. To explore further, just check out Electoral Calculus and play with their prediction facility.

In fact, that is not quite right, because uniform swings don't really predict Lib Dem performance. In 1983, 25 per cent of the vote got them 23 seats, whereas in 2005, 22 per cent got 62 seats: a vote increase of 1 per cent in 2010 saw a seat decline of five. The point? On 8 to 10 per cent, which is where they are currently in the polls, they could as easily face wipeout as a doubling in their representation to c.16-20 seats: it all depends on their ability to focus support.

From the outset, though, I was sceptical of the May landslide scenario. It always needed just three things to happen for that result to look very unlikely indeed:

• Tory vote dipping below 42 per cent
• Labour recovering to 36 per cent
• Lib Dems hanging on (whatever their actual support levels) to 12-plus seats

This is precisely what the latest polls are suggesting. Factor in, also, that many polling organisations, having underestimated the Tory vote in 2015, are now correcting estimates upward, which raises concerns that they might now be overcorrecting.

Run those figures through Electoral Calculus, fixing Lib Dems at around 12-15 seats, and you will see why there are rumours of Conservative jitters. Because, after all this, they could end up with the same – or fewer – seats as now.

How likely is that? And how significant are polls suggesting Labour has slashed the Conservative lead?

The view from David Butler, the man credited with inventing the science of psephology (the statistical study of elections and trends in voting), is that uncertainty is the order of the day. On Newsnight last week, he argued that the polls were volatile to a degree he had not seen in 50 years of analysing elections. Really? Or is he, too, being sucked in by media excitement over recent polls.

The statistician's view, represented by Nate Silver (who did a better job of predicting the US 2016 election than many credit), warns us to be cautious when interpreting outliers – any observation point significantly distant from other observations. This is because outliers can represent true variability in the underlying data or some sort of skew or bias within the measurement process, and it is impossible to know until after the event.

This is compounded twice over in election polls. First, their sheer quantity: whenever the same question is tested multiple times using different samples, different methodologies, statistics warn us that forecasts will vary – sometimes wildly. Second, because they may represent genuine shift. Polls at odds with polls from last week might indicate one set was inaccurate, or it might just mean that voting intentions have genuinely shifted.

Silver's view, as relevant today as when he wrote about the subject back in 2012, can be summarised as: don't panic! Neither discount outliers entirely – don't throw out data without very good reason – nor place too much weight on them.

Interrogate the data coming to us from each poll, looking for systemic bias. But unless absolutely necessary, don't exclude odd data, but roll it into the running average. Seek out trends, and seek to back those with independent data sources, such as economic indicators or the state of the economy. Relevant here, for instance, is YouGov positivity trackers showing Corbyn's rating soaring to +26 over the last weeks, while May's has dipped to -59.

But don't obsess about them nor, he warns, news organisations that claim big trends on the back of exceptional polls

A good example of this is an interesting analysis, from Buzzfeed special correspondent James Ball, arguing that the UK is back to two-party politics.

He makes a good case for that being so in this election. Is it, though, as various media then proclaimed, a "trend"? To be fair to Ball, he appears, like myself, agnostic on this. Two-party politics may be back for 2017, but whether for the long term, only time will tell.

How, then, should we be using polls to make sense of what is happening? The first key piece of advice is: don't! Elections are complex beasts and attempting to analyse them through the lens of polling, with its ifs and buts and 3 per cent margins of error, is like using a telescope to read a newspaper – alternately too focused and not focused enough.

But if you must, look for trends, look for consistency. Use rolling averages, rather than aggregates. The first will help detect real trends, the second often gives disproportionate weight to early polling.

Use polls to inform scenarios, rather than provide direct forecasts. On the basis of current polling here's my own personal hostage to fortune:

If polls keep moving against the Conservatives, a hung parliament is possible – in which case, expect short-term turmoil, and May overtaking Sir Alec Douglas-Home as the shortest serving Prime Minister in modern times. He made it to 363 days. By June 9 she will have served just 331 days.

Otherwise, expect business as usual with a more or less chastened Conservative Party continuing in power. ®