Analyzing open-ended survey questions: the “Essential Guide” Framework
The twists and turns of today’s dynamic consumer landscape means it has never been more important to have your finger on the pulse of evolving consumer preferences and behaviors, and what’s driving them. Insights from closed-ended questions tell you a lot about the consumer, but often missing is a deeper narrative around “why.” This is where analysis of open-ended survey questions comes in, allowing you to start with a blank canvas (no pun intended) that gets painted by unfiltered responses relatively free of question bias. The answers to these questions help you better understand consumer preferences, attitudes, and feelings, as well as identify additional opportunities for further in-depth analysis. This post will present a simple framework for analyzing open-ended survey questions.
Through our work helping hundreds of organizations get more value from open-ended text, we’ve developed an analysis framework for open-ended survey questions. This framework can be applied across different types of research based on the objectives and structure of the particular study as outlined in our eBook Core Insights: The Essential Guide to Analyzing Open-Ended Questions.
Framework for analyzing open-ended questions
To generate core insights, analysis of open-ended questions needs to go beyond simply counting the words and phrases, or indicating binary sentiment (i.e. positive/negative). Researchers want to know what the respondent means, and, if possible, why. A combination of three related lenses of analysis can be used to unpack the meaning and narrative of open-ended text: thematic analysis, emotion measurement, and correlation.
Thematic analysis of unstructured text
Thematic text analysis is the process of parsing the ideas or things that are expressed in text and presenting those ideas in a structured way. The kinds of questions we’d approach with thematic analysis will be the ones where respondents are answering “what” questions, such as “what was the main idea in this episode?” Or “what brands come to mind when you think of shaving cream?” Thematic analysis is a broader version of topic analysis. Asking the respondent to describe their “perfect ice cream treat” might include responses like “Tillamook,” “Ben & Jerry’s,” “mint chocolate chip,” “local,” or “waffle cone.” These could be considered and scored individually as topics, but likely take on more meaning when organized into broader, hierarchical categories, like “brands,” “flavors,” “serving types,” and “other qualities.” Many researchers will recognize this hierarchical approach as equivalent to a code frame. While basic topic analysis may suffice in some studies, like unaided brand questions, this second level of thematic analysis generates additional, potentially more nuanced insights.
Emotion analysis (aka Sentiment Analysis)
Emotion analysis is an extrapolation of how the respondent “feels” based on how they have expressed themselves. This is a more nuanced version of sentiment analysis, which typically categorized statements into positive, negative and neutral (more on this below). Generally “how” or “why” questions are more likely to generate an emotional reaction vs. “what” questions. Examples include questions like “How do you feel about the last season of the show?” and “Why did you give that rating?” Naturally, some topics are inherently more likely to generate emotional reactions vs. others. For example a bank asking “What did you like best about your experience?” is likely to register a different level of emotion vs. an amusement park asking the same question.
There are several popular emotion measurement frameworks that can be applied to open-ended text. The most basic of course is a binary sentiment analysis of positive and negative (plus neutral). While the simplicity of this can be helpful, modern language, what Canvs calls the “unnatural language” of social media, makes discerning positive and negative challenging (particularly for AI). “The brand is dope” might be interpreted as positive, while “Politicians are dopes.” might be seen as negative. We prefer a judgment-neutral framework that identifies discreet emotions. The Eckman framework utilized 7 basic emotions of anger, contempt, disgust, enjoyment, fear, sadness, and surprise, while the Plutchik model identifies eight primary emotions: anger, anticipation, joy, trust, fear, surprise, sadness and disgust. Canvs has developed a proprietary, 42-emotion model for emotion measurement to provide additional nuance for researchers, but also allows researchers to switch frameworks and compare between them. Whether conducted by a person or an AI, the “art” of emotion measurements is in the interpretation of language and applying that interpretation in a consistent, unbiased way.
Correlation analysis of text
Correlation analysis is the ability to identify statistically relevant connections between information from the open-ended text and associated data from the close-ended questions, such as respondent demographic cross-tabs. For example, you may find that a specific age range prefers one type of nut butter vs. another, respondents with different income levels have different preferences in their “perfect ice cream treat,” or that women describe an ad creative with different emotional words vs. men. You could also find connections between different attributes of a product described in the open-ended text and the emotional reaction (i.e. correlation between two or more data from the open ends themselves).
Applying These Lenses to Market Research
These analytical approaches to open-ended questions can be applied slightly differently across various types of research, largely driven by the topic and objectives of the project.
Consumer behavior research
Thematic analysis in consumer behavior studies allows researchers to identify the top ideas expressed about the subject, as well as to group and quantify respondents that express similar themes, whether those themes are emotional, topical, or broad categories.
Utilizing nuanced emotion measurement (vs. general sentiment) can help identify the level of emotional intensity around a subject. Some questions, such as “How do you feel about inflation?” are designed specifically to unlock emotion measurement.
Correlation analysis is often used to identify meaningful differences among sub-segments of the sample (eg. demographic cross-tab analysis).
Brand and awareness studies
Naturally, in Unaided questions, thematic analysis is used first and foremost to identify the specific brands mentioned. In association questions, thematic analysis will identify the attributes driving emotional response. Researchers can track KPIs of awareness overall using unaided awareness questions, or breakthrough of certain messages based on a campaign (association of ‘family’ with a bank brand, for example).
While Unaided Awareness questions are, by design, non-emotional, Brand Association questions can elicit extreme and nuanced emotions. Researchers can use this approach to understand what attributes and themes drive “annoying” versus “hate”, or “enjoy” versus “love.”
Beyond demographic and brand-specific cross-tabs, correlation analysis can be used to identify the emotional resonance of common brand associations or understand how a brand is resonating across demographic segments.
Thematic analysis in product research is used to identify the attributes perceived and valued by the respondent. Feedback captured as part of sensory lab setting may be looking for the impact of certain product choices or formulations within the open-ended responses. Attributes are often summarized into thematic categories, such as “taste” and “texture” in food research, often referred to as codes (the attributes) and nets (the broader themes).
Researchers often measure the emotional reaction against established benchmarks.
Beyond looking at demographic cross-tabs, correlation analysis can be used in product research to connect attributes with emotional engagement. For example, respondents who’s responses indicate “Love” may be more likely than others to identify a specific product attribute (eg. rich, Corinthian leather).
Customer experience / NPS surveys
Thematic analysis is used to identify and categorize specific aspects of customers’ experience that naturally arise in the comments. These can be both positives (essentially, what do people like moste) and negatives (what are people unhappy or complaining about).
Emotional analysis can provide color to your NPS scores. Identifying what topics & themes drive emotions, and can provide actionable feedback for teams hoping to improve their NPS scores.
Perhaps the most compelling use of correlation analysis in customer satisfaction research is the use of open ends to identify potential root causes of customer satisfaction/ dissatisfaction. Essentially “what topics are more likely to be mentioned by promoters/detractors” in the NPS lexicon. Additionally, correlating emotional engagement with specific elements of the experience can also be useful.
Ads & creative testing
In ad and creative testing, thematic analysis is used to measure message effectiveness and recall without pre-selected options that might bias the response. Additionally, specific creative elements can be identified (“what did you like best about…”).
Analysis of open ends in creative testing is likely to start with sentiment. How did the viewer feel about the ad or show? In many cases, researchers measure overall emotional reaction (% of comments with emotion) and specific emotional reactions (love, hates, funny, etc.). Over time, benchmarks can be developed to measure emotional reaction against norms.
One of the most compelling uses of correlation analysis is the ability to identify the topical drivers of emotional reaction (i.e. correlation of topics and emotions). Additionally, demographic cross-tabs are critical in a field where cultural relevance is a primary success factor.
Product reviews & other non-survey analysis
Thematic analysis can be used to identify specific topics and broader themes emerging from non-survey sources of open-ended text. Given the wide range of potential comments, researchers are more likely to focus on the top themes vs. exhaustive, long-tail analysis.
Non-survey open ends can be mined as a source of unfiltered consumer sentiment. As with ad testing, overall emotional engagement can be benchmarked to identify meaningfully elevated (or depressed) emotional reactions.
Without access to weighted demographics for cross-tab analysis, correlation analysis will typically focus within the text itself in terms of how consumers feel and what’s driving those feelings.
Get the essential guide to open ends
We’ve packaged up this framework, along with some great open-ended question examples, in a conveniently downloadable eBook titled Core Insights: The Essential Guide to Analyzing Open-Ended Questions.