The News
First came the AI polling respondents. Now come the AI-assisted pollsters.
Most of us don’t like taking surveys, and when we do, we usually want them to be over as quickly as possible. OpenResearch, the OpenAI-funded research nonprofit, wanted to see if using AI chatbots as part of the process could make respondents more engaged and entice them to offer fuller responses.
It turns out the answer is yes. And that could be a big deal for the survey business.
OpenResearch tested the theory in its ongoing study of unconditional cash transfers, which looks at how people respond to getting regular payments, with one group receiving $1,000 a month for three years and a control group getting $50 a month. It checked in with them regularly, through online surveys and with more in-depth phone interviews, even after the payments ended. And in its latest online survey on stress, it offered respondents the opportunity to chat with a bot.
More than three-quarters have taken up the offer so far (the survey is ongoing). Those who opted to talk to the bot spent a median time of 16 minutes each on the survey, often offering more comprehensive answers that would not have been possible with a multiple-choice format. More importantly, the bot could probe respondents’ answers and ask them to clarify or expand, giving OpenResearch a much richer set of data. Two-thirds of the respondents rated the experience as positive, and a similar percentage said they would be willing to do it again.
In this article:
Know More
Traditional polls have gotten a bad rap in recent years, not least after they seemed to miss the level of Donald Trump’s support in 2016. As landlines have dwindled and do-not-call lists have proliferated, it’s been harder and harder to get respondents on the line.
Meanwhile, online surveys have taken up the slack, but pollsters remain constrained by how many questions people will answer and how much time they’ll spend on a survey.
In-depth, qualitative surveys can dig deeper into issues, but researchers simply don’t have the time or money to pay an army of interviewers.
AI chatbot-assisted surveys open up the prospect of blending aspects of both qualitative and quantitative surveys at scale. People may be more willing to engage with questions if the system they’re interacting with seems responsive to their answers; at the same time, bots could draw out more nuance and context from a respondent’s answers by asking follow-up questions.
There are other advantages as well — writing survey questions is its own art form, requiring pollsters to carefully parse language that may or may not mean the same thing to every respondent. AI-assisted polling could help researchers build systems that adapt to responses, rather than be forced to impose a rigid uniformity on them. Asking if someone is employed, for example, might elicit a range of answers that include everything from steady work to being part of the gig economy to volunteering. “Before AI, we had to ask the same questions in exactly the same way,” said Elizabeth Rhodes, OpenResearch’s research director. AI-assisted systems could allow researchers to focus more on what they want to find out, rather than how to word questions.
To be sure, there are also challenges — fuller and more nuanced answers can also be harder to sort and categorize, and there still needs to be more work done to establish standards and ethical frameworks, noted Rhodes.
But overall, the potential of AI to help researchers better understand and analyze public opinion seems enormous.
Gina’s view
Much of the immediate thinking about how generative AI will disrupt industries has focused on gaining efficiencies or taking away rote work. OpenResearch’s experience — and that of Aaru, a company we profiled last year — shows that the real disruptions will come in new capabilities that in many cases we haven’t envisioned yet. True, human pollsters could have, in theory, engaged in long conversations with every respondent they reached. But in practice, that simply wasn’t possible. Chatbots change that dynamic, and potentially, the entire business.
It’s not just that qualitative research at scale may become possible; it’s that polling could become a much more dynamic process, with questions adapting to responses in much shorter time frames.
There’s another potential dynamic at play, too: Will humans be more willing to divulge more sensitive details to a machine than another human? Three-quarters of the respondents said they were comfortable enough with the interaction to honestly describe their experiences with stress.
At Harvard’s Shorenstein Center, Fellow Shuwei Fang describes what she calls “the intimacy dividend” — the potential that people might welcome the opportunity to speak frankly and freely with a chatbot without the possibility that another human might judge them. (OpenResearch tried to program in some level of empathy into the chatbot’s responses; the reactions were mixed. The organization suggests later experiments might try to customize the bot’s tone to each participant.)
Room for Disagreement
On the other hand, about a fifth of the respondents said they felt that interacting with the AI system was “creepy.”