How Did the Polls Underestimate Trump?

Trump WTF.jpg

The day after Donald Trump’s US Presidential Election victory The Dominion Post ran a headline saying ‘WTF’. It left off the question mark so not to cause offence (and asked us to believe that they really meant ‘Why Trump Flourished’). But the question lingers regardless.

For those of us in the research industry, the question we have been asked most often since then – and the question we have asked ourselves most often – is ‘how did the polls get it so wrong?’. It’s a good question. And coming hot on the heels of the polls’ failure to predict Brexit, an important one.

People have tried to answer this question in a number of ways, and each of them tells us something a little different about the nature of polling, the research industry, and voters in general.

The first response might be called the ‘divide and conquer’ argument. This is the one that says not all the polls got the election result wrong. The USC/LA Times poll, for instance, tracked a wave of support for Trump building and predicted Trump’s victory a week out. Similarly, the team at Columbia University and Microsoft Research also predicted Trump’s victory. But what this argument doesn’t do is explain why most polls clearly got it wrong (and even a broken clock is right twice a day).

There is a variation on this argument that we might call ‘divide and conquer 2.0’. This is the argument that says people outside of the industry misunderstood what the polls actually meant. The best example here might be Nate Silver’s Before the election 538 gave Trump about a thirty percent chance of winning. To most people, that sounds like statistical short hand for ‘no chance’. But to statisticians, it means that if we ran the election ten times, Trump would win three of them. In other words, Silver was saying all along that Trump could win. Just it was more likely that Hilary would. As Nassim Nicholas Taleb might put it, the problem here is that non-specialists were ‘fooled by randomness’. So the problem isn’t with the pollsters but the pundits.

The next argument might be called ‘duck and run’. This is the argument that says the fault lies with the voters themselves because they probably misrepresented their intentions. Pollsters typically first ask people if they intend to vote, and only then who they’re going to vote for. But, of course, there’s no guarantee the answer to either is accurate. This seems to be the explanation that David Farrar (who is one of New Zealand’s most thoughtful and conscientious pollsters) reached for when approached for comment. Given how many Americans didn’t vote in the election, expect to hear this argument often.

A variation on this ‘duck and run’ argument is that polls are at their least effective where a tight race is being run. In this election, nearly 120 million votes were cast but the difference between the two candidates was only about 200,000 (or less than one third of one percent). It could be that no polling method is sufficiently precise to work under these conditions. If you want to try this line of argument in the office, award yourself a bonus point for referring to the ‘bias-variance dilemma’.

But I think all of these arguments are a kind of special pleading. Worse than that, much of what the industry is now saying looks like classic hindsight bias to me. This is also known as the ‘I-Knew-It-All-Along Effect’, which describes the tendency, after something has happened, to see the event as having been inevitable (despite not actually predicting it). While it’s easy to be wise after the fact, the point of polling is to provide foresight, not heroic hindsight.

And no matter how well intentioned any of these arguments might be, it’s hard not to think we’ve seen them all before. Philip Tetlock’s masterful Expert Political Judgment: How Good Is It? reports a 20 year research project tracking predictions made by a collection of experts. These predictions were spectacularly wrong but even more dazzling was the experts’ ability to explain away their failures. They did this by some combination of arguing that their predictions, while wrong, were such a ‘near miss’ they shouldn’t count as failure; that they made ‘the right mistake’; or that something ‘exceptional’ happened to spoil their lovely models (think ‘black swans’ or ‘unknown unknowns’). In other words, the same arguments that we’re now seeing the polling industry rolling out to explain what happened with this election.

For me, all of these arguments miss the point and distract us from the real answer. The pollsters (mostly) got the election wrong because the future – despite all our clever models and data analytics – is fundamentally uncertain. Our society loves polls because we crave certainty. It’s the same reason we fall for the Cardinal Bias, the tendency to place more weight on what can be counted than on what can’t be. But certainty will always remain out of reach. What Trump’s victory really teaches us is that all of us should spend less time reading polls and more time reading Pliny the Elder. It was Pliny, after all, who told us ‘the only certainty is that nothing is certain’.

How Did the Polls Underestimate Trump?

When To Accentuate The Positive?

Doing it wrong

Recently one of our researchers presented at a conference for PR and Communications professionals and highlighted the importance of ‘loss aversion’ in human behaviour. This describes how all of our brains are wired to experience losses much more acutely than gains. As a result, our researchers suggested that when they wanted to influence behaviour, communications professionals should talk about the costs of not doing something rather than the benefits of doing it.

In the question time following that presentation, one of the conference participants noted that this idea flies in the face of conventional communication practice which places the emphasis on the positive message. So which is it?

Fortunately, social science has a clear answer. According to Peter Salovey, it depends on whether the new behaviour we want to promote is perceived as risky or safe. If the person we’re talking to considers the new behaviour to be safe, the key is to emphasise all the good things that will happen if they change to it.

But where they believe the new behaviour is a risk, the challenge is to overcome the status quo bias. To do this, we need to emphasise the bad things that will happen if they don’t change. This makes taking that risk more appealing, because of the threat of that loss.

So the lesson seems to be to accentuate the positive where the audience sees safety, and emphasise the negative where they fear risk.


When To Accentuate The Positive?

Shakespeare, on Impact Measurement


Is your work making an impact?

This seems like such an obvious question to ask but answering it is fraught with difficulty. One of the problems is that ‘impact’ can be such a slippery concept. Even setting aside for a moment the question of what counts as an ‘impact’, measuring impact means being clear about such things as:

  • Impact for whom? Where? When?
  • How much of an impact?
  • How long did the impact last?
  • Was it worth it? (i.e., did the scale and duration of the impact justify the investment and effort?) .

The key to being able to successfully measure impact is to be clear at the outset what it is you are setting out to achieve. In other words, before you cry havoc and engage your cogs of awe, you need to be clear about what success looks like. Nor is it enough for you (and your team) to be clear about what success looks like – you need to write it down so you can refer to it later.

Once you know what it is you want to achieve, you can then work on your theory of change. This is simply a logical diagram that outlines how you are going to achieve the success you’ve clearly outlined.

Let’s say you work in communications and PR and you want to know if you’re work is making a difference. Drawing a simple logic model will take you from your work in communications to the impact that you want to create. But what is useful about this kind of logic model is that it clearly distinguishes between things like activities (what you do); outputs (the things you create); and impact (the success you want to achieve):

Sytems logic 2

These distinctions are critical because they help us resist the understandable urge to look in the wrong places and count the wrong things. Too much communication evaluation has focused on outputs and outcomes precisely because they are easy to see and simple to measure (indeed, for many analytical reports both are automated).

But outputs and outcomes are not impacts. More critically, they may not even be reliable markers on the way to impact. Think about this example: You’ve been asked to create a campaign to get people out of their cars and onto buses. Tapping into your genius for communication, you and your team create a multi-level and multi-channel approach built around a series of catchy messages. Because the campaign’s been carefully crafted to have an attention-grabbing gonzo element, the whole thing goes viral, is covered in the mainstream media, and wins your agency an illustrious award. Time for tea and medals for everyone?

Probably not. No matter how hard and brilliantly you work (‘activities’), no matter how clever the campaign materials are (‘outputs’), and no matter how often links are clicked, stories are read, and your client is interviewed on Paul Henry’s show (‘outcomes’), all of that is for nought if people don’t actually get out of their cars and onto buses.

As Shakespeare said about something else, ‘ambition should be made of sterner stuff’. Which is why it is important to be clear about success before the awards start rolling in and Paul Henry starts calling your client. There’s nothing wrong with building a buzz, but for this campaign you can’t claim to be driving change until people change how they drive.

If you talk to your marketing colleagues about this they will nod as sagely as a tree full of owls. That’s because marketers are taught in Stage One classes that customers don’t go to hardware stores to buy drills but to buy the holes those drills make. They are also taught that there is no point talking about the features of you product (‘a drill with extended battery life’) if you don’t know the benefits the customers want. The same logic applies to communications.

In other words, we need to think carefully (and deeply) about what we’re trying to do before we reach for any kind of measurement tool. The notion of ‘evidence-based’ (or ‘evidence-led’) approaches to communication practice is an attractive one, but we first need to be clear about what we’re trying to measure with evidence. There are over 150 measurement tools in the TRASI (Tools and Resources for Assessing Impact) database but none of them are any use if we keep looking in the wrong place.

Or as Shakespeare put it, without understanding impact, evaluation of communication effectiveness will remain a ‘tale told by an idiot, full of sound and fury, signifying nothing’.


Research First is PRINZ’s research partner, and specialises in impact measurement, behaviour change, and evidence-based insights

Shakespeare, on Impact Measurement

Managing Waiting Times


It should be no surprise that most people find waiting (and delays) aversive. It is also no surprise that waiting (and delays) are inevitable. What is a surprise is how poorly this time is managed.

David Maister wrote about how to better manage waiting times back in the mid-1980s. His paper, The Psychology of Waiting Lines, showed how the key to giving customers a much better waiting experience was in understanding why waiting is experienced so negatively. Maister showed that uncertainty makes waiting seem longer (and that anxiety compounds this uncertainty, making the wait seem even longer); that unanticipated and unexplained waits are worse; that occupied time feels shorter; that unfair waits (where some people get to jump the queue) are much more aggravating than equitable waits; and that solo waits seem longer than group or social waits.

The implications for anyone having to manage customers who need to wait are obvious: communicate clearly about waiting times, provide certainty that the service will be delivered at the end of their wait, and find things for them to do while they wait.

But the real insight from Maisters’ work, though, is that the psychology of queuing is more important than how long is spent waiting. As a result, there are real customer experience gains to be made in manipulating the queue so the waiting time feels shorter. Houston Airport provides a masterclass in how to do this but no-one does it better than Disney.



Managing Waiting Times

Why Is It So Hard To Get Anything Done?


If you ever get to the end of the day wondering why you haven’t achieved anything that you set out to do, ‘Interruption Science’ might have the answer.

As the name suggests, Interruption Science is the study of how interruptions affect our performance. What this science reveals is that interruptions don’t just reduce our performance, they ravage it.

Researchers at the University of California, Irvine, found that it took an average of 23 minutes for workers to return to their prior level of performance following an interruption. Research from the University of Michigan shows that even very short interruptions can seriously diminish performance. In that study participants who were interrupted for just three seconds were twice as likely to make a mistake on their original task as those who were not interrupted.

Another useful insight from Interruption Science is that we all get interrupted more than we probably realise. W Edwards Deming claimed that the average American worker experienced fifty interruptions a day but things seem to have gone downhill since then. One estimate is that we are now interrupted every three minutes. But even if that is an exaggeration, you can see the real problem here: when you combine the number of times we are interrupted with the time taken to recover from interruptions, it’s no surprise that so many of us feel we are getting nothing worthwhile accomplished.

Given this, is it any wonder that we try to fit the ‘real work’ into those times when we are on our own or have the office to ourselves? Count yourself in this group if you find yourself working late (or arriving early) so you can work in peace, or if you take work home to get it finished.

The bad news is that you can’t escape the interruption performance trap by working long hours. At least not for very long. There is a large body of evidence that shows long hours of work end up hurting your productivity (and your health). Similarly, as I have written about previously in this blog, multitasking can’t square the circle because it is mostly an illusion. As Dilbert might put it, ‘multitasking is the single best way to screw up both jobs’.

Instead of trying to work around interruptions, we need to find ways to contain them. The best way to do this is to structure your day around blocks of time where you can focus on key tasks and not be interrupted. I like the idea of buying some ‘do not disturb’ signs off eBay and using them to let everyone else in the office know when you need to be left alone (some of those ‘Quiet Please’ signs you see on golf courses would be better still).

Alternatively, you could try sharing this article with your boss to help them understand how fewer interruptions benefit everyone (you might add that research in the USA argues that workplace interruptions cost that economy about US$500 billion a year in lost productivity). With your boss on-board, you can create a timesheet code for ‘head-down’ time which everyone in your office can use to work without interruption.

It might also pay to try to schedule this head-down as early in your working day as you can. While the jury is still out on this, there is some evidence from behavioural science that the first two hours of the working day are when we should be at our most productive (but that ability is often squandered with frequent interruptions).

If you struggle to get your employer to understand the value creating dedicated interruption-free working time, the next best approach might be to simply say ‘no’ more often. The key here will be making everyone understand that you will be more productive and effective if you are able to discriminate between the things that do and don’t need your attention. And if you need some help learning to say no, try rolling out the Polish proverb that says ‘Not my circus. Not my monkeys’.

There are also some simple things you can do to reduce interruptions. The most obvious is to turn off the alerts on your phone, email, and social media feeds. Another great insight from Interruption Science is that we are just as likely to interrupt ourselves as we are to be interrupted by someone else. If you’ve ever stopped working on something to check your Trademe auction, or look at Facebook update, you’ll know what I mean. Try reducing how often you do that and you might be surprised what you can get done in a day.

Why Is It So Hard To Get Anything Done?