Using AI to Interpret Survey Data

Stop pasting interview transcripts into ChatGPT and asking for a summary. You’re not getting insights—you’re getting blabla. Here’s how to actually extract signal from qualitative data with AI. A lot of product teams are experimenting with AI for user research. But most are doing it wrong. They dump all their interviews into ChatGPT and ask: “Summarize these for me.” And what do they get back? Walls of text. Generic fluff. A lot of words that say… nothing. This is the classic trap of horizontal analysis: → “Read all 60 survey responses and give me 3 takeaways.” → Sounds smart. Looks clean. → But it washes out the nuance. Here’s a better way: Go vertical. Use AI for vertical analysis, not horizontal. What does that mean? Instead of compressing across all your data… Zoom into each individual response—deeper than you usually could afford to. One by one. Yes, really. Here’s a tactical playbook: Take each interview transcript or survey response, and feed it into AI with a structured template. Example: “Analyze this response using the following dimensions: • Sentiment (1–5) • Pain level (1–5) • Excitement about solution (1–5) • Provide 3 direct quotes that justify each score.” Now repeat for each data point. You’ll end up with a stack of structured insights you can actually compare. And best of all—those quotes let you go straight back to the raw user voice when needed. AI becomes your assistant, not your editor. The real value of AI in discovery isn’t in writing summaries. It’s in enabling depth at scale. With this vertical approach, you get: ✅ Faster analysis ✅ Clearer signals ✅ Richer context ✅ Traceable quotes back to the user You’re not guessing. You’re pattern matching across structured, consistent reads. ⸻ Are you still using AI for summaries? Try this vertical method on your next batch of interviews—and tell me how it goes. 👇 Drop your favorite prompt so we can learn from each othr.

I tried to compete with AI on data analysis 🤖 Shocker, I lost. Here's what happened... Out of habit, I started analyzing hundreds of detailed responses to an annual survey we send the team after our company retreats. Halfway through, I realized I should have leaned on AI to assist with this 🤦🏻♂️ so I decided to use this as an opportunity to measure the time saved on this routine task. After 7 hours of meticulous manual analysis, I asked Claude (Anthropic's AI) to do the same task. It took 15 minutes to get the same results 😅 Even more impressive, I compared the quality of both analyses, and Claude's was better! It caught the exact same major themes but also spotted patterns I'd missed, probably because I had my own biases about what worked/didn't work. This is probably common sense for many of you now, but just in case, here's how to replicate this process: - Send a feedback form to your team (Google Forms, Typeform, etc) - Export the responses (CSV works best for me) - Upload the file to your AI tool of choice (I use Claude) - Ask it to: Summarize the common themes, list top things that worked well, list areas for improvement, identify data trends, create a TLDR, etc - Share insights with your team The key takeaway for me... Many of us are still adapting to the power of the tools we have at our disposal, and I often find it easy to fall back into doing manual work out of habit or perhaps a bias toward my capabilities. But the use of AI in the modern workplace isn't about replacing human work - it's about complementing it. And it's tasks like these that provide the perfect opportunity to leverage the power of AI so we can focus our energy on implementing the conclusions it helps us create. This was a good reminder for me, hope it's helpful for some of you as well! **Photo below from this retreat in Ireland - stay tuned for another post with more details from the survey and what we learned from this retreat.

If you work in #CME, can you use AI to summarize free-form written responses to evaluation survey questions? Yes. But in the interests of content integrity, here are some things to bear in mind. ➡️ AI lacks the ability to understand context and nuance that is essential for valid qualitative analysis. It can miss underlying themes, perspectives and emotions. ➡️ AI summaries are not a substitute for methodical qualitative coding and analysis. They are probabilistic labels, not analytic themes derived from a systematic framework. ➡️ There are transparency and reproducibility issues with AI-generated insights that undermine content integrity and validity. The rationale behind AI's decisions is a black box. ➡️ Biases in the AI's training data can get amplified and reproduced in the generated summaries, leading to misrepresentations. Pairing AI with established qualitative analysis software can help ground the findings. I use MaxQDA and Nvivo. Cautiously experiment with AI, but don't view it as a replacement for rigorous qualitative methods. Read my full article in the Alliance For Continuing Education in the Health Professions Almanac.

📢 New publication alert! 📢 I'm thrilled to share our recent article in the Journal of Leadership Studies, "Prompting for Meaning: Exploring Generative AI Tools for Qualitative Data Analysis in Leadership Research"! Co-authored with Creighton University Adjunct Professor Shannon Cleverley-Thompson, Ed.D., and University of Southern Maine Ph.D. students Dan Erikson, Anna Blankenbaker, and Brooke Brown-Saracino, this study explores how generative AI (GenAI) tools like ChatGPT, Claude, and NotebookLM can be used for qualitative data analysis in leadership research. We piloted a three-way comparison methodology with graduate students, who performed AI-assisted analysis and compared the results with both expert human coding and their peers' work. Key Takeaways from Our Research 🔬 GenAI as a collaborative partner: We found that GenAI can support various phases of qualitative research, like identifying themes and patterns, but it requires human oversight for interpretive depth, ethical considerations, and bias detection. Students learned to view AI as "a second set of eyes" rather than a replacement for human analysis. The Power of Prompting: Our students' prompting strategies evolved from simple, quantity-focused queries to more intentional, values-driven and context-aware prompts6. This practice, which we call "prompting as stewardship," helps maintain discernment and direction when guiding AI tools, ensuring a balance between efficiency and interpretive control. Addressing AI Anomia: The study's three-way comparison framework fostered what we call "productive epistemic friction". This process helps students resist the tendency to accept AI outputs as authoritative and, instead, question what might be missing or oversimplified. It prepares them to navigate AI environments by developing the critical discernment needed to identify and address "AI Anomia," a term for when vague or euphemistic language masks a delegation of responsibility to individuals who lack the authority or context to govern these systems. Our findings show that integrating GenAI thoughtfully through pedagogical frameworks that emphasize human-AI collaboration can enhance analytical rigor and prepare emerging researchers to leverage technology while maintaining the interpretive richness essential to qualitative inquiry. Additional thanks to our editorial team for this issue, Christine Haskell, Erik Bean, Ed.D., Tashieka S. Burris-Melville, EdD, Jimmy Payne, CNP, Ph.D., and Vijayanth Tummala, Ph.D.! Read the full article here: https://lnkd.in/eff-p4Qp

During the last few weeks, I have spoken with many UX colleagues about their concerns regarding the use of AI. The two issues that consistently come up are hallucination and inconsistency. People worry that one model produces one set of themes, another model generates slightly different conclusions, and suddenly the analysis feels unstable and unreliable. These concerns are valid, however I believe they are partially manageable. Hallucination often happens when a model is asked to generate insights without grounding in actual data. One of the most effective ways to reduce this risk is using Retrieval Augmented Generation, or RAG. Instead of allowing the model to rely on its general training patterns, RAG forces it to retrieve relevant interview segments first and then generate insights only from those retrieved passages. When every theme must be anchored to specific verbatims, unsupported claims become far less likely. Inconsistency across models does not necessarily indicate failure. In fact, it can be used strategically. In traditional qualitative research, we rely on multiple human coders. We assess agreement, examine disagreement, and refine our categories accordingly. The same logic can be applied to AI. Running two different models in parallel for thematic analysis acts as a form of inter rater reliability. Each model independently extracts themes grounded in retrieved evidence. Then we compare them. Do they converge on similar clusters? Do they reference overlapping verbatims? Do they assign similar structural roles to the same behavioral patterns? When both models converge, confidence increases. When they diverge, that signals ambiguity, boundary issues, or data complexity. Disagreement becomes a diagnostic signal rather than a weakness. This is where Bayesian analysis adds another layer of rigor. Instead of stopping at agreement percentages, we can formally quantify uncertainty. We can estimate the posterior probability that a theme is truly prevalent given evidence from multiple models. We can model how strongly certain themes predict outcomes such as churn intention or satisfaction. We can update those probabilities as more interviews are collected. Rather than saying a theme appears important, we can estimate how likely it is to dominate across segments with credible intervals that reflect uncertainty. 1-AI provides scale and pattern detection. 2-RAG provides grounding and traceability. 3-Parallel models provide triangulation. 4-Bayesian analysis provides formal uncertainty modeling. When these components are combined thoughtfully, qualitative AI analysis shifts from a fragile black box to a structured probabilistic system. The real transformation is not about using AI faster. It is about designing AI workflows that are auditable, triangulated, and statistically grounded.

Struggling to make sense of customer feedback? Here’s how AI can help. If you’ve ever felt overwhelmed by a pile of testimonials, reviews, or survey responses, you’re not alone. Most small business owners know there are insights in there… but don’t have time to dig them out. That’s where AI tools like ChatGPT and Gemini come in. Here’s how to use them to quickly find patterns, improve your messaging, and understand what really matters to your customers: Collect your feedback Export your Google reviews, email testimonials, or survey responses into one document. It doesn’t have to be perfect — just copy and paste. Ask AI to summarise themes Prompt example: 🗣️ “Can you identify the top 3 strengths and 3 weaknesses mentioned in these customer comments?” You’ll get a quick snapshot of what’s working (and what’s not). Dig deeper into emotions and language Prompt example: 🗣️ “What language or phrases do customers use when describing why they chose us?” Use these phrases in your website copy or ads — it's literally your customers telling you what resonates. Look for objections and concerns Prompt example: 🗣️ “Are there any common objections, frustrations or hesitations mentioned in these reviews?” You can then address these in your FAQs, emails, or onboarding flow. You don’t need to be a tech expert. You just need to ask good questions. If you’re already using AI for content, try pointing it at your feedback. You might be surprised at what you learn. #aimarketing #chatgpt #gemini

AI can now answer surveys for us. And it's surprisingly accurate. I read a paper recently that made me rethink how we approach consumer research. They tested this against 57 real consumer surveys with 9,300 human responses. SSR achieved 90% of human test-retest reliability. This reminded me of my internship at a major FMCG company. Part of my project involved consumer research. It took weeks to get approvals, hiring a research agency, user empanelment, analytics, and thousands of dollars to interview a few users. The traditional approach is slow and expensive. A survey panel for a national product launch can cost tens of thousands of dollars and take weeks to complete. An SSR-based simulation can deliver comparable insights in hours. One finding from the paper surprised me: Human respondents tend to be polite and often tell you what you want to hear. The AI panel produced wider, more discriminative signals between good and mediocre concepts. How you can use this for your marketing: If you want to test messaging angles, product positioning, or creative concepts before committing a budget, there's now an entire category of synthetic research platforms. Here are some tools which are using this technique and are worth exploring: Synthetic Users, Delve AI, Yabble, Quantilope, and Artificial Societies. The paper is worth reading if you are curious about the methodology. It opens up interesting possibilities for teams that need faster iteration cycles on messaging and positioning. – I share weekly posts on AI, marketing, and the tools changing how we work. Follow along if you find this useful.

Another Alteryx + Anthropic Claude #AI use case! This one focuses on #HR surveys and using LLMs to aggregate and distill complex text feedback. Employee feedback has historically been a challenge. All of the 1-10 multiple-choice stuff is easy. Comments are challenging and where much of the valuable input/feedback is hidden. Solutions like this can help with sentiment analysis, categorization, summarization, etc. You could also alert HR/Legal if sensitive topics need to be handled immediately or in a compliant way. Narrative feedback can be in yearly surveys, exit surveys, company comment boxes, etc. In the most uplifting case, you can use this to share all of the amazing success and employee satisfaction with the company. In the most serious cases, you can catch comments about safety, impropriety, hostile work environments, etc. This type of analysis can also help with expense reporting, where comments are required to determine whether the expenses fit within your company's policies. The #LLMs like #Claude, #Gemini, #Llama, #OpenAI, #Grok, etc, are opening up functionality that was virtually impossible before. Capitalize Analytics believes AI has a massive part in our future, but we also know that technology hype requires real-life uses before it becomes valuable to companies. We are constantly looking for ways to use these technologies to solve our clients' challenges. It has to be a lot more than just "cool." Let us know what you think. Are these examples helpful?

The numbers shocked me. I put the new advanced AI model from ChatGPT to the test. We just closed down our annual marketing study. There were more than 700 responses to one important question. The big question: "If you've considered attending Social Media Marketing World 2025, but have not yet registered, please tell us why." There were a wide range of text responses. If I got this analysis wrong, it could result in marketing messages that fail to convert. So I decided to run an experiment. I put Claude 3.5 Sonnet head-to-head against ChatGPT's latest model—o3-mini-high. Both AI tools were asked to analyze the open-ended responses. I then asked them to quantify the results. What happened next was eye-opening... ChatGPT claimed there were only about 200 responses. That wasn't even close—it missed nearly 500 responses! But Claude? It nailed it. Not only did it correctly identify all 700+ responses, but it also grouped them perfectly into 7 distinct categories. It even created a dynamic chart showing the patterns in seconds—something that would've taken my team days to do manually. The clarity was stunning. This wasn't just about counting responses. This was about understanding real human feedback at scale—and making important marketing decisions based on that data. And one AI tool clearly dominated the other. It got me thinking about how many marketers are using the wrong tools for their analysis. Are you using AI to analyze your customer feedback? Which tool works best for you? I'd love to hear from you.

When it comes to AI-powered market research, it's time to challenge the conventional wisdom. Replicating human survey results is often seen as the gold standard, but what if that's not enough? Traditional surveys, and even naive AI models, tend to overstate consumer intentions, missing the mark on real-world actions. Through an experiment with Ask Rally's language models, we found that a basic model replicated survey biases (78% of simulated responses favored an eco-friendly car), yet switching to a more advanced model cut this figure to 37%, much closer to actual market behavior. The takeaway? The true advantage lies not in mirroring traditional methods but in choosing and calibrating AI models that bridge the intention-action gap. This approach not only aligns synthetic research with reality but could redefine how we predict consumer behavior altogether.