Talk to the bot: When to use an AI interviewer
Sometimes not being human is a good thing.
AI-powered moderator tools are gaining lots of attention in the research world - hoping to help us unlock a new superpower: speed, speed, speed.
Providers promise to streamline the process from start to finish to allow you to interview at scale. These tools can create discussion guides, conduct interviews, analyze data, and generate shareable reports all on their own.
Within the Research and Data Science (RAD) team at Intercom, we’ve been testing these tools and found that their pros and cons are oddly similar.
Pro: They’re not human
AI moderation proved especially useful in circumstances where an interviewee would rather not speak to a person, like after they’ve churned. Automation can make interactions feel low-stakes and encourage users who may be hesitant to provide feedback.
AI tools are also highly scalable and can conduct dozens of interviews simultaneously. With a low barrier to entry, you can get quick answers to precise questions.
Con: They’re not human
AI moderators can’t quite replicate the nuance of reciprocal engagement and struggle to pick up on interesting threads that come up in conversation. Any researcher knows these oftentimes lead to the most important takeaways from an interview and help to uncover new areas of examination. AI moderators are quick, but not necessarily quick on their feet as they’re ultimately limited by the guidance and context we provide them.
Overall, AI moderation is most beneficial when both sides of the interview favor an efficient, frictionless exchange over rapport. In these instances, using AI can facilitate easy, worthwhile insights.
Based on our experiences, here are some use cases for AI moderation and where we’d strongly favour human-led research.
Where AI moderation works well
Speaking with disengaged teams
We used this pilot to speak with team members from lost accounts, and this approach drove more traction than any previous churn-related research initiative. Typically, we see a 1.5% response rate but this jumped to 4% when we were clear the session would be AI-moderated and could be conducted at any time.
Customers are often nervous to engage in post-churn calls, it can be hard to tell a business why you left them or why their product wasn’t good enough. And no matter how many times you use the classic ‘research purposes only’ tagline, there’s always a fear the session will drift into a sales call.
With AI moderation, interviews are quick and don’t require someone to show up on a live call, eliminating the fear of being judged or put on the spot. In these cases, AI may actually offer a more guaranteed “safe space” for people to be candid about their experiences.
Running a pilot
AI interviews can be great when you’re not quite sure how to structure your research yet. A conversation or two with customers can often help you pressure-test your research plan, but the overhead of scheduling and moderating might stop you from going that route.
Using an AI moderator to run a pilot is a quick, low effort way to figure out if your framing will land, if your interview structure will work and if you’re generally thinking in the right way. You’ll get directional signal without adding an extra few days to your project plan.
Continuous learning
AI-moderated interviews work well in continuous learning contexts, when you have a fairly clear sense of the topic and the specific areas you want to dig into.
The tool allows you to be explicit in your prompts about where the interviewer should probe and what threads they should follow, so having an existing framework for how these types of conversations go will help to get focused, relevant insights.
Where AI moderation doesn’t work
Early stage discovery
AI-moderated interviews are much harder to use in true discovery mode when you’re still figuring out what the problem even is.
Without a framework or guide for the interviews, an AI interviewer doesn’t know what to ask. In these moments, the value of research comes from following surprising tangents, reading between the lines, and knowing when to throw the discussion guide out entirely.
There’s also something important about internalizing early-stage insight to start building your own intuition. AI moderation can surface themes, but it doesn’t yet help you develop the deep understanding you need to become the expert in a new problem space.
Long-form interviews
From a participant perspective, a long interview with an AI moderator can feel both tiring and transactional. Without the natural energy and responsiveness of a human on the other end, it’s easier to tune out – especially over longer sessions.
For research that relies on building rapport, maintaining momentum, or adapting in real time to what someone is sharing, a human moderator is more engaging and effective.
Analysis and synthesis
While AI analysis tools are helpful for summarizing what happened, we wouldn’t rely on them for analysis or synthesis.
It’s difficult to layer in additional context (e.g. company priorities, recent discussions or decisions, etc.) that shapes how we analyse research in the moment. That context is critical for making insight actionable, and it’s something we typically think about during analysis, not upfront.
Today, AI-generated outputs are most useful as a reference point: a way to cross-check your own take, or to get a high-level recap of what’s been said. They can support you as you’re trying to piece together your point of view, but shouldn’t lead the way.
Speaking with our customers
We know our customers value giving feedback and feeling like that feedback is genuinely heard. A 1:1 conversation shows that we’re investing real time and attention, whereas AI moderation doesn’t convey that same sense of care.
That said, we can see a place for offering AI-moderated interviews as an option – especially for customers with busy schedules, in different time zones, or who simply prefer async formats.
Striking a balance
AI moderation certainly has its place in the research toolkit.
When the goal is to bypass social friction or scale structured feedback, it is often the ideal solution. Used intentionally, these tools allow us to move faster and hear from those who are otherwise hard to reach. But there are (many) moments when your own intuition and empathy are needed to drive the most valuable research sessions - especially when we don't yet know what we don't know.
 |
|