The pollsters Waterloo

“…Ladd made one criticism that will have a familiar ring today: “In recent years, election polls have frequently overestimated the Democratic share of the vote. Since the 1980s, exit pollsters have tended to overstate Democratic candidates’ strengths.” “Frequently” may have been too strong, but the problem was and is significant. A Pew Research Center analysis of 2020 polls conducted after September 15, 2020, found that “93 percent of national polls overstated the Democratic candidate’s support among voters.” Eighty-eight percent of polls did so in 2016….”

Karlyn Bowman, "The pollsters’ Waterloo,"Forbes, November 18, 2021, https://www.aei.org/articles/the-pollsters-waterloo/

The pollsters’ Waterloo

Twenty-five years ago today, The Wall Street Journal excerpted an article written by the political scientist Everett Carll Ladd with the title, “The Pollsters’ Waterloo.” Ladd didn’t pull any punches. “Election polling,” he said, “had a terrible year in 1996. The entire enterprise should be reviewed...” The reaction to the article from some pollsters was swift and severe. Some clearly saw his criticisms as a threat to their livelihood, while others said his description of final poll results on which he based his criticism was erroneous. A quarter century later, however, the pollsters are thriving, but the criticisms Ladd raised continue to cast a shadow over their work.

Ladd made one criticism that will have a familiar ring today: “In recent years, election polls have frequently overestimated the Democratic share of the vote. Since the 1980s, exit pollsters have tended to overstate Democratic candidates’ strengths.” “Frequently” may have been too strong, but the problem was and is significant. A Pew Research Center analysis of 2020 polls conducted after September 15, 2020, found that “93 percent of national polls overstated the Democratic candidate’s support among voters.” Eighty-eight percent of polls did so in 2016.

Ladd urged the pollsters to convene a blue-ribbon commission like their predecessors did after the polling debacle in the 1948 election. After all, as the late Andrew Kohut, one of the giants in the field who was a principal at both Gallup and Pew said, the survey business should not be judged solely on the performance of election polls, but these polls are the “most visible expression of the validity of the survey research method.” In both 2016 and 2020, under the auspices of the American Association for Public Opinion Research (AAPOR), a distinguished group of practitioners and academics reviewed publicly available national and state-level results. The big takeaway from the new report: the polling error in 2020 “was the highest in 40 years for the national popular vote and the highest in at least 20 years for state-level estimates of the vote in presidential, senatorial, and gubernatorial contests.” The polling error that favored Biden over Trump was consistent across different survey modes.

In 1948, the reviewers concluded that the pollsters stopped conducting their polls too early, and that voters made up their minds late in that campaign. In 2016, the AAPOR task force pointed to several factors: a large number of late deciders who broke disproportionately for Donald Trump and incorrect weighting for voter education levels in many key states. The 2020 report ruled out those factors and a number of others.

The 2020 task force could not say with certainty given available data why polls overstated the Democratic margin. But they advanced a number of possible explanations. One is that Trump’s strongest supporters simply chose not to respond to pollsters, while those who did voted more Democratic, thereby skewing results. They speculated that nonresponse could have contributed, as could weighting measures and a large infusion of new voters.

If it is a key explanation, the nonresponse problem may be specific to Trump. His criticisms were harsh, but they certainly weren’t new. Others have tried to undermine the poll participation as well. In 1984, the popular Chicago Tribune columnist Mike Royko urged his readers to join him in what he called a “noble pursuit,” to “lie” to the pollsters in the Illinois primary that year. And after the 2000 election, a blue-ribbon panel, with honorary cochairs Gerald Ford and Jimmy Carter, “strongly encourage citizens not to participate in exit polling.”

Ladd lodged other criticisms as well: he believed the sheer volume of pre-election polling activity and relentless media coverage of the horse race distorted elections. As a friendly critic, I would add other criticisms. With the possible exceptions of Gallup and Pew, pollsters incessant chasing of headlines and focus on polarization has caused them to abandon many old questions that gave us a fuller sense of who we are. They have given short shrift to ordinary life in favor of politics, politics, politics and the negativity it conveys. Pollsters shouldn’t abandon political coverage. It is essential. But other subjects should be given their due.

Ladd was a long-time friend and mentor. Despite his criticism, it was always clear to me that he wanted the survey business, which he loved and thought vital to our understanding of America and democracy, to perform better. He would no doubt appreciate the impressive efforts of the 2016 and 2020 task forces. He would applaud the extraordinary work pollsters have done on COVID. But he would be saddened that there is neither a clear answer to a problem he identified a quarter century ago, nor an obvious way to fix it, nor an effort to provide a fuller picture of American life.