New study shoots down popular Democratic theories

A large-scale new study of political advertising will challenge some popular Democratic Party theories about how to win elections, and argue that advertising for everything from presidential candidates to English muffins should be based more in experimentation than in theory.

The peer-reviewed study, set to be released soon in the American Political Science Review, examined 146 experiments on 617 advertisements Democratic campaigns produced in 2018 and 2020. The ads were tested with 500,000 survey respondents on a research platform called Swayable. The study’s authors then asked independent political scientists to tag the advertisements by elements of their style and substance in what appears to be the largest randomly-controlled test of American political ads ever conducted.

The puzzling finding: Some ads were markedly more successful than others, but there was “no persistent pattern to what worked best,” according to a presentation on the data by Swayable co-founder and CEO James Slezak, who is one of the study’s authors.

In particular, the study offers challenges to the various corners of a Democratic Party stuck in a long-running argument between populists and popularists, identity politicians and class warriors – all of them seeking to direct the vast torrents of money that flow through various committees.

A couple of the surprising non-findings:

— “Popularism” – the obvious-seeming notion that campaigns should focus on positions that poll well and avoid ones that poll badly – didn’t clearly win out in the data. “Issue choice was not a reliable predictor of what ads persuade voters,” the study found according to Slezak’s presentation, which said spots focused on racism didn’t turn off viewers as some had predicted.

— Catering to identity politics didn’t consistently work, either. The “identity of narrators didn’t generally impact persuasion much.” The study also found that “voters of all backgrounds were comparably persuadable” and responded similarly to the same messages.

The paper’s findings are unlikely to settle the question of how much politicians should shape their campaigns around the results of public opinion surveys. But at the very least, it suggests that keeping an eye on polling isn’t always a reliable shortcut to producing a compelling ad.

“This is a much more rigorous spotlight on the questions that get debated” in campaign advertising, said Michael Podhorzer, a central figure in Democratic politics who has seen the paper’s results. “I’ve spent a gazillion dollars on politics and made this kind of decision,” said Podhorzer, who recently retired as political director of the AFL-CEO, a longtime major political advertiser. “The truth is that nothing gets a big reaction in any predictable way.”

The Swayable data is both a shot across the bow of current center-left political wisdom and a pitch for a more tightly-optimized version of big money politics. And its underlying point is both obvious and annoying to people who talk professionally about politics: That public opinion is a complex moving target, and that what worked last week might not work next.

American presidential campaigns are among the world’s biggest advertisers, quadrennially dumping billions of dollars into television, digital, radio, and mailings, which dominate campaign expenditures. David Ogilvy’s much-quoted line that “half the money I spend on advertising is wasted, and the trouble is I don’t know which half,” is now 60 years old. Bottom-of-funnel click-to-buy ads on Google or email fundraising are disturbingly optimized.

And yet when it comes to real-world actions, like voting and shopping, the business of data is rife with questionable statistics and, worse, outright charlatans, like the figures who turned Cambridge Analytica into a household name with false claims about its role in Donald Trump’s 2016 victory.

Slezak co-founded Swayable 2017 with Valerie Coffman, now the company’s CTO, whom he’d met when they were both getting their PhDs in experimental physics from Cornell University. Slezak chose persuasion after science, was a founding partner of the agency Purpose and then worked for 3 years in senior digital strategy roles at The New York Times.

The pair’s company, which went through the startup accelerator Y Combinator in 2018, offers to add speed and statistical rigor to the process of “testing” alternate versions of advertisements with voters or consumers, an age-old exercise that often centers on focus groups and consultants’ instincts.

Slezak said the field reminds him of his time at Cornell, when theorists lined up outside experimentalists’ laboratories to test their conjectures.

And while he is the latest in a long line of entrants in pitching more rigorous and scientific market research than predecessors, he’s particularly disdainful of some of the big voices in Democratic politics right now, which he describes as “the lab coat hustle.”

“They are presenting themselves as empiricists but they aren’t behaving that way,” he says of proponents of popularism. “It’s a cargo cult – ‘I’m doing the things they say empirical people do.’”

Swayable has won converts in the Democratic party: The firm worked for the Democratic Congressional Campaign Committee in 2020 and 2022, and will lead that committee’s message testing efforts in 2024. It also lists commercial clients including Amazon and Meta. A senior brand manager for Thomas’ English muffins, Mike Jensen, told me last week that he’d been dropping rough ads into Swayable because “in food, grocery, it’s hard for us to sometimes link our marketing back to sales.”

(The APSR study is not yet public. I reviewed it on the condition I only quote from Slezak’s presentation, whose claims track its peer-reviewed statements. The study’s lead authors are Luke Hewitt, whose MIT PhD was based on data science he did for Swayable; University of California, Berkeley professor David Broockman, Yale’s Alexander Coppock and Ben Tappin, a postgraduate at MIT. The other authors are Slezak, Coffman, former Obama aide Nathaniel Lubin, and Mohammad Hamidian.)

The study’s details aren’t public, putting critics at a disadvantage. But I asked the writer Matthew Yglesias, a leading proponent of popularism, for his reaction to its argument.

He responded with skepticism rooted in “the obvious point that ‘I have a miracle solution to make your group’s ideas win in politics’ is a much better sales pitch for a company than ‘sorry the best thing is for candidates to say/do what’s popular.’”

• “Democrats should do a lot of polling to figure out which of their views are popular and which are not popular, and then they should talk about the popular stuff and shut up about the unpopular stuff,” Ezra Klein paraphrased in his profile of popularist anaylst David Shor in 2021.
• Popularism “was important across the 2020 cycle because so many influential leaders in the progressive community took actions that were foolish and counterproductive,” Yglesias wrote last year.
• The Democratic analysts’ divisions are social as well as political, Politico noted last year in a review of dueling cocktail parties.