'A black eye': Why US political polling missed the mark – again
WASHINGTON (NYTIMES) – Senator Susan Collins did not lead in a single publicly released poll during the final four months of her re-election campaign in Maine. But Ms Collins, a Republican, won the election comfortably.
Senator Thom Tillis, a North Carolina Republican, trailed in almost every poll conducted in his race. He won, too.
And most polls underestimated President Donald Trump’s strength, in Iowa, Florida, Michigan, Texas, Wisconsin and elsewhere.
Instead of winning a landslide, as the polls suggested, Mr Joe Biden beat Mr Trump by less than 2 percentage points in the states that decided the election.
For the second straight presidential election, the polling industry missed the mark. The miss was not as blatant as in 2016, when polls suggested Mr Trump would lose, nor was the miss as large as it appeared it might be on election night.
Once all the votes are counted, the polls will have correctly pointed to the winner of the presidential campaign in 48 states – all but Florida and North Carolina – and correctly signalled that Mr Biden would win.
But this year’s problems are still alarming, both to people inside the industry and to the millions of Americans who follow presidential polls with a passion once reserved for stock prices, sports scores and lottery numbers. The misses are especially vexing because pollsters spent much of the last four years trying to fix the central problem of 2016 – the underestimation of the Republican vote in multiple states – and they failed.
“This was a bad year for polling,” David Shor, a data scientist who advises Democratic campaigns, said.
Douglas Rivers, the chief scientist of YouGov, a global polling firm, said: “We’re obviously going to have a black eye on this.”
The problems spanned the public polls that voters see and the private polls that campaigns use. Internal polls, conducted for both Democratic and Republican candidates, tended to make Republican candidates look weaker than they were.
“District-level polling has rarely led us – or the parties and groups investing in House races – so astray,” David Wasserman of the Cook Political Report, a nonpartisan publication that analyzes races, wrote last week.
The full explanation for the misses will not be knowable for months, until the election results are finalised, detailed poll data is released by survey firms, and public voter files – showing exactly who voted – are also released.
Once this information is available, some of the immediate postelection criticism of polls may end up looking even worse than the polls themselves, academic researchers caution.
But the available facts already point to some likely conclusions:
– People’s decreasing willingness to respond to polls – thanks partly to caller ID – has reduced average polling response to only 6 per cent in recent years, according to the Pew Research Centre, from above 50 per cent in many polls during the 1980s. At today’s level, pollsters cannot easily construct a sample of respondents who resemble the population.
– Some types of voters seem less willing to respond to polls than others, perhaps because they are less trusting of institutions, and these voters seem to lean Republican.
– The polling industry tried to fix this problem after 2016, by ensuring that polling samples included enough white working-class voters in 2020. But that is not enough if response rates also vary within groups – for instance, if the white or Hispanic working-class voters who respond to polls have a different political profile than those who do not respond.
– This year’s polls may have suffered from pandemic-related problems that will not repeat in the future, including a potential turnout decline among Democratic voters who feared contracting the coronavirus at a polling place.
– A much-hyped theory that Trump supporters lie to pollsters appears to be wrong or insignificant. Polls did not underestimate his support more in liberal areas, where supporting Mr Trump can be less socially acceptable, than in conservative areas.
– In what may be the most complex pattern, polls underestimated the support of multiple Senate Republican candidates even more than Mr Trump. This means the polls missed a disproportionate number of Americans who voted for both Mr Biden and a Republican Senate candidate – and that the problems do not simply involve Trump’s base.
Regardless, there are reasons for concern: Polling now seems to suffer from some systemic problems, which create a misleading picture of the country’s politics.
This problem has sprung up at the same time that a deeply polarised country has become more intensely interested in politics than it once was. The share of eligible voters who turned out this year may have reached the highest level since 1900.
Book sales about politics have soared, as have ratings for television news and subscriptions to publications that cover politics. Many people crave polls, which can have an addictive quality, especially during an election that both parties described as an existential battle for America’s future.
Politics has become a high-stakes spectator sport at the same time that the country’s ability to understand it has weakened.
Pollsters have been grappling with some of the same challenges since the creation of the industry in the early 20th century.
One of the first polls to receive widespread attention came from The Literary Digest, a magazine, and it was published days before the 1916 presidential election. The magazine asked readers in 3,000 communities to mail in sample ballots and then reported that President Woodrow Wilson was in a stronger position than his Republican opponent, Charles Evans Hughes.
Wilson won the election, and The Literary Digest poll became a national phenomenon, correctly pointing to the winners from 1920 through 1932. In 1936, though, the poll showed that Alf Landon, the Republican nominee, would easily defeat Franklin D. Roosevelt. Instead, Roosevelt won in a landslide, and the error changed the industry.
Although Literary Digest’s sample of 2.4 million respondents was enormous, it was not representative. The magazine’s circulation skewed toward affluent Americans, who were more hostile to Roosevelt and the New Deal than most voters. A less prominent pollster that year, George Gallup, had surveyed many fewer people – about 50,000 – but he had been careful to ensure they matched the country’s demographic mix. Gallup correctly predicted a Roosevelt win.
Gallup’s methods shaped the industry, which today consists of dozens of organisations, with a wide range in quality. Typically, a poll surveys hundreds of or a few thousand people and then extrapolates their answers to represent the broader population. If a poll cannot reach enough people in a certain demographic group – say, white Catholics or older Black men – it counts those it does reach in the group more heavily.
Still, the Gallup methods were not perfect, for some of the same reasons that polls have struggled recently. Within some demographic groups, Gallup turned out to be interviewing more Republican voters than Democratic ones, and it overestimated the Republican vote share in 1936, 1940 and 1944.
In 1948, the pollsters’ luck ran out. They continued to overestimate the Republican vote share, reporting throughout the campaign that President Harry Truman trailed his Republican opponent, Thomas Dewey. This time, the election was close enough that the polls pointed to the wrong winner, contributing to perhaps the most famous error in modern journalism, the Chicago Tribune’s banner headline “Dewey Defeats Truman.”
That error led to a new overhaul of polling. Pollsters redoubled their efforts to build samples that were representative of the country. They were not perfect: Presidential polls in the 1950s and 60s missed by about 4 percentage points on average, similar to this year’s miss, according to the website FiveThirtyEight.
“Polling has always been challenging,” Nate Silver, FiveThirtyEight’s editor-in-chief, said.
But polls still tended to point correctly to the winner of the presidential race in those years, partly because Americans were so willing to respond to polls.
People tended to answer their telephone when it rang. They also tended to trust major institutions, like the government, the media and higher education. And they did not have a constant source of entertainment in their pocket – a smartphone – that an extended telephone survey kept them from using, as Mr Shor, the data scientist, said.
“Decades ago, most people would be happy to answer the door to a stranger or answer the phone to a stranger,” Courtney Kennedy, the director of survey research at Pew, said, “and those days are long gone.”
A Shock in 2016
When response rates began falling in the 1980s and 1990s, many people in the polling industry worried that they were facing a crisis: How could they accurately measure Americans’ opinions if many refused to respond to surveys?
Then came the election of 2016.
Nearly every poll showed Hillary Clinton to be leading Trump in Michigan, Pennsylvania and Wisconsin. The leads were big enough that her campaign paid relatively little attention to those states. But she lost all three narrowly, giving Trump a stunning victory.
Afterward, pollsters dug into their data, comparing it to precinct-by-precinct election results and to voter files, which show who voted but not how they voted. In early 2017, the leading industry group, the American Association of Public Opinion Research (AAPOR), released its conclusions.
One factor was largely unsolvable: Late-deciding voters, accurately identified as undecided in polls, broke strongly for Trump. Many may have been swayed by James Comey, the FBI director, who nine days before the election sent a letter to Congress announcing that he was again looking into Clinton’s use of a private email server as secretary of state.
A second factor was more damning for pollsters. Many had not ensured that their samples included enough people without college degrees, especially among white voters.
Perhaps most important, the polling association argued, the 2016 experience did not suggest a systematic problem in which polls favored one party. In some years, like 2012, polls slightly underestimated the Democratic share, and in other years, like 2016, they slightly underestimated the Republican share. The report said the direction of those misses was “essentially random.”
The midterm elections of the following year, 2018, initially seemed to support this conclusion. The polls correctly suggested that Democrats would sweep to victory in the House, while Republicans would retain the Senate. State polls were off by an average of about 4 percentage points, which was historically normal.
The underlying details contained some reasons for concern, though. While polls in some liberal states, like California and Massachusetts, had underestimated the Democrats’ vote share in 2018, polls in several swing states and conservative states, including Florida, Georgia, Michigan and Pennsylvania, again underestimated the Republican share.
For the second time since Trump’s entry in politics, the polls had somehow failed to reach enough Republican voters in the swing states that decide modern presidential elections. A third election – his reelection campaign – was looming in 2020, and it was one that millions of Americans, both his supporters and critics, would be following passionately.
Missing Republican Voters
By the final weeks of this year’s campaign, the polls seemed to be telling a clear story: Biden had led Trump by a significant margin for the entire race, and the lead had widened since the summer. Some combination of the coronavirus, Trump’s reaction to police brutality and his erratic behaviour at the first debate had put Biden within reach of the most lopsided presidential win since Ronald Reagan’s in 1984.
In the campaign’s final 10 days, a Wall Street Journal/NBC News poll showed Biden up by 10 percentage points nationwide, as did a YouGov poll. Fox News put the lead at 8 points, and CNN at 12. Other polls also showed the lead to be at least 8 points.
In Wisconsin, which Mr Trump had won narrowly in 2016, a YouGov poll found Mr Biden up by 9 percentage points. A New York Times/Siena College poll showed the lead was 11 points, while Morning Consult said 13 points. A Washington Post/ABC News poll reported the lead was 17 points. In congressional races, the Cook Political Report called the Democrats clear favorites to retake the Senate and gain seats in the House.
That, of course, did not happen. Mr Biden won Wisconsin by less than 1 percentage point, while Republicans fared much better in congressional races than polls had suggested.
What went wrong? There are almost certainly multiple causes, pollsters and political scientists says. One possibility is that the pandemic may have led to an unexpected falloff in Election Day voting among Democrats, given that the party emphasized mail voting. Another is that Democratic voters, energized by the Trump presidency and bored during the pandemic, became newly excited to respond to polls.
But the most likely explanation remains an unwillingness among some Republican voters to answer surveys. This problem may have become more acute during Trump’s presidency, because he frequently told his supporters not to trust the media.
“I think when all the votes are counted, what we are going to see is a far smaller polling error, potentially even minimal, in many of the states where the presidential was competitive,” Jefrey Pollock, a Democratic pollster and president of the Global Strategy Group, said. But he acknowledged that some polls were off and added, “As professionals, we have to question whether a segment of the electorate has opted out of talking to us.”
In some ways, the problem is new. It is a reflection of modern technology, political polarisation and more. In other ways, though, the problem has existed since the 1930s, when polls also undercounted segments of the working-class vote.
One difference is that those undercounted voters leaned Democratic at the time, which led polls to understate the strength of Roosevelt and Truman. The partisan effect has since flipped, with the white working class now backing Republicans, but the underlying dynamic has remained the same.
Pollsters managed to fix the problem after their “Dewey Defeats Truman” reckoning. The question is how they can do it again now, when survey response rates have fallen well below 1 per cent.
The Media’s Polling Problem
Perhaps the one pollster who has emerged from the last few years with the best reputation is J. Ann Selzer, who runs a firm in West Des Moines, Iowa, and who conducts polls with The Des Moines Register.
One of her methods, she said in an interview, is keeping her surveys short, because there are differences between voters who are willing to talk at length to a pollster and those who are not. Most of her surveys last less than 15 minutes. In her final survey before an election, she tries to keep the interviews under eight minutes.
“There’s a self-selection in people’s willingness to talk to polls,” she said. She recalled conducting a 45-minute-long survey for a private client years ago about Transcendental Meditation. “Our finding was that about half the people we talked to had an experience with Transcendental Meditation,” she said. “Do you think that’s true?” The polling industry group, AAPOR, had announced months before the election that it would conduct a post-mortem analysis for 2020. This analysis – and others, done by individual pollsters – will probably shape the specific measure that pollsters take.
Regardless, there is unlikely to be any single step that fixes the polls’ recent anti-Republican lean. Instead, pollsters are likely to try a mix of many small measures, like Selzer’s short interviews.
One option is to create new screening questions about whether respondents trust other people and major institutions – and then weight less trustful respondents more heavily in a poll’s final results. Pew, in recent years, has asked questions about whether people spend time volunteering, as one measure of trust.
Another is to expand the use of text messages and other nonverbal communication, like Facebook messages, in surveying people. “We’re going to see more diversity in polling methodologies,” said Kevin Collins, the co-founder of Survey160, which collects data through text messaging.
FiveThirtyEight’s Silver, then an independent blogger, created a breakthrough in 2008 when he began writing about every available poll, focusing on the swing states and talking about the probability of one candidate beating another. Before that, most publications had focused on national polls and largely ignored those done by competing publications.
The New York Times published Silver’s blog, FiveThirtyEight, from 2010 through 2013, and it is now part of ABC News. Other publications have since taken a similar approach, creating their own probabilistic models.
Virtually nobody thinks polling is going away. It is too important in a democracy, Collins, of Survey160, said. It guides campaign strategies and politicians’ policy choices. And there is no alternative method of election analysis with anywhere near as good a track record as polling’s imperfect record.
The only short-term solution, some people believe, is for pollsters and the media to emphasise – and for Americans to recognize – that polling can be misleading. Even an aggregate picture, from dozens of polls, can be meaningfully off, especially in an intensely divided political era.
Source: Read Full Article
