Don't miss Jessica Bates from National Geographic and Trip Kucera from Canvs during TMRE Continued | Dec 6th at 12:10 PM ET

How to Analyze Open-Ended Survey Questions – A Framework

6 min read

| September 30, 2022

Conducting analysis on open-ended survey questions 

The twists and turns of today’s dynamic consumer landscape means it has never been more important to have your finger on the pulse of evolving consumer preferences and behaviors, and what’s driving them. Insights from closed-ended questions tell you a lot about the consumer, but often missing is a deeper narrative around “why.” This is where open-ended questions come in, allowing you to start with a blank canvas (no pun intended) that gets painted by unfiltered responses relatively free of question bias. The answers to these questions help you better understand consumer preferences, attitudes, and feelings, as well as identify additional opportunities for further in-depth analysis. 

Through our work helping hundreds of organizations get more value from open-ended text, we’ve developed an analysis framework to organize responses. This framework can be applied across different types of research based on the objectives and structure of the particular study. This framework anchors the approach captured in our eBook Core Insights: The Essential Guide to Analyzing Open-Ended Questions.

 

Open-ended question analysis frameworks 

To generate core insights, analysis of open-ended questions needs to go beyond simply counting the words and phrases, or indicating binary sentiment (i.e. positive/negative). Researchers want to know what the respondent means, and, if possible, why. A combination of three related lenses of analysis can be used to unpack the meaning and narrative of open-ended text: thematic analysis, emotion measurement, and correlation. 

Thematic analysis 

Thematic text analysis is the process of parsing the ideas or things that are expressed in the text and presenting those ideas in a structured way. The kinds of questions we’d approach with thematic analysis will be the ones where respondents are answering “what” questions, such as “what was the main idea in this episode?” Or “what brands come to mind when you think of shaving cream?” Thematic analysis is a broader version of topical analysis. Asking the respondent to describe their “perfect ice cream treat” might include responses like “Tillamook,” “Ben & Jerry’s,” “mint chocolate chip,” “local,” or “waffle cone.” These could be considered and scored individually as topics, but likely take on more meaning when organized into broader, hierarchical categories, like “brands,” “flavors,” “serving types,” and “other qualities.” Many researchers will recognize this hierarchical approach as equivalent to a code frame. While basic topic analysis may suffice in some studies, like unaided brand questions, this second level of thematic analysis generates additional, potentially more nuanced insights. 

Emotion analysis 

Emotion analysis is an extrapolation of how the respondent “feels” based on how they have expressed themselves. Generally “how” or “why” questions are more likely to generate an emotional reaction vs. “what” questions. Examples include questions like  “How do you feel about the last season of the show?” and “Why did you give that rating?” Naturally, some topics are inherently more likely to generate emotional reactions vs. others. For example a bank asking “What did you like best about your experience?” is likely to register a different level of emotion vs. an amusement park asking the same question. 

There are several popular emotion measurement frameworks that can be applied to open-ended text. The most basic of course is a binary sentiment analysis of positive and negative (plus neutral). While the simplicity of this can be helpful, modern language, what Canvs calls the “unnatural language” of social media, makes discerning positive and negative challenging (particularly for AI). “The brand is dope” might be interpreted as positive, while “Politicians are dopes.” might be seen as negative. We prefer a judgment-neutral framework that identifies discreet emotions. The Eckman framework utilized 7 basic emotions of anger, contempt, disgust, enjoyment, fear, sadness, and surprise, while the Plutchik model identifies eight primary emotions: anger, anticipation, joy, trust, fear, surprise, sadness and disgust. Canvs has developed a proprietary, 42-emotion model for emotion measurement to provide additional nuance for researchers, but also allows researchers to switch frameworks and compare between them. Whether conducted by a person or an AI, the “art” of emotion measurements is in the interpretation of language and applying that interpretation in a consistent, unbiased way. 

Correlation analysis

Correlation analysis is the ability to identify statistically relevant connections between information from the open ends and associated data from the close-ended questions, such as respondent demographics. For example, you may find that a specific age range prefers one type of nut butter vs. another, respondents with different income levels have different preferences in their “perfect ice cream treat,” or that women describe an ad creative with different emotional words vs. men. You could also find connections between different attributes of a product described in the open-ended text and the emotional reaction (i.e. correlation between two or more data from the open ends themselves). 

 

Applying These Lenses to Research

These analytical approaches to open-ended questions can be applied slightly differently across various types of research, largely driven by the topic and objectives of the project.

Consumer behavior research

Thematic Analysis

Thematic analysis in consumer behavior studies allows researchers to identify the top ideas expressed about the subject, as well as to group and quantify respondents that express similar themes, whether those themes are emotional, topical, or broad categories.

Emotion Analysis

Utilizing nuanced emotion measurement (vs. general sentiment) can help identify the level of emotional intensity around a subject. Some questions, such as “How do you feel about inflation?” are designed specifically to unlock emotion measurement.

Correlation Analysis

Correlation analysis is often used to identify meaningful differences among sub-segments of the sample (eg. demographic cross-tab analysis).

Brand and awareness studies

Thematic Analysis

Naturally, in Unaided questions, thematic analysis is used first and foremost to identify the specific brands mentioned. In association questions, thematic analysis will identify the attributes driving emotional response. Researchers can track KPIs of awareness overall using unaided awareness questions, or breakthrough of certain messages based on a campaign (association of ‘family’ with a bank brand, for example).

Emotion Analysis

While Unaided Awareness questions are, by design, non-emotional, Brand Association questions can elicit extreme and nuanced emotions. Researchers can use this approach to understand what attributes and themes drive “annoying” versus “hate”, or “enjoy” versus “love.”

Correlation Analysis

Beyond demographic and brand-specific cross-tabs, correlation analysis can be used to identify the emotional resonance of common brand associations or understand how a brand is resonating across demographic segments.

Product research 

Thematic Analysis

Thematic analysis in product research is used to identify the attributes perceived and valued by the respondent. Feedback captured as part of sensory lab setting may be looking for the impact of certain product choices or formulations within the open-ended responses. Attributes are often summarized into thematic categories, such as “taste” and “texture” in food research, often referred to as codes (the attributes) and nets (the broader themes).

Emotion Analysis

Researchers often measure the emotional reaction against established benchmarks.

Correlation Analysis

Beyond looking at demographic cross-tabs, correlation analysis can be used in product research to connect attributes with emotional engagement. For example, respondents who’s responses indicate “Love” may be more likely than others to identify a specific product attribute (eg. rich, Corinthian leather).

Customer experience / NPS surveys 

Thematic Analysis

Thematic analysis is used to identify and categorize specific aspects of customers’ experience that naturally arise in the comments. These can be both positives (essentially, what do people like moste) and negatives (what are people unhappy or complaining about).

Emotion Analysis

Emotional analysis can provide color to your NPS scores. Identifying what topics & themes drive emotions, and can provide actionable feedback for teams hoping to improve their NPS scores.

Correlation Analysis

Perhaps the most compelling use of correlation analysis in customer satisfaction research is the use of open ends to identify potential root causes of customer satisfaction/ dissatisfaction. Essentially “what topics are more likely to be mentioned by promoters/detractors” in the NPS lexicon. Additionally, correlating emotional engagement with specific elements of the experience can also be useful.

Ads & creative testing

Thematic Analysis

In ad and creative testing, thematic analysis is used to measure message effectiveness and recall without pre-selected options that might bias the response. Additionally, specific creative elements can be identified (“what did you like best about…”).

Emotion Analysis

Analysis of open ends in creative testing is likely to start with sentiment. How did the viewer feel about the ad or show? In many cases, researchers measure overall emotional reaction (% of comments with emotion) and specific emotional reactions (love, hates, funny, etc.). Over time, benchmarks can be developed to measure emotional reaction against norms.

Correlation Analysis

One of the most compelling uses of correlation analysis is the ability to identify the topical drivers of emotional reaction (i.e. correlation of topics and emotions). Additionally, demographic cross-tabs are critical in a field where cultural relevance is a primary success factor.

Product reviews & other non-survey analysis 

Thematic Analysis

Thematic analysis can be used to identify specific topics and broader themes emerging from non-survey sources of open-ended text. Given the wide range of potential comments, researchers are more likely to focus on the top themes vs. exhaustive, long-tail analysis.

Emotion Analysis

Non-survey open ends can be mined as a source of unfiltered consumer sentiment. As with ad testing, overall emotional engagement can be benchmarked to identify meaningfully elevated (or depressed) emotional reactions.

Correlation Analysis

Without access to weighted demographics for cross-tab analysis, correlation analysis will typically focus within the text itself in terms of how consumers feel and what’s driving those feelings.

Get the essential guide to open ends 

We’ve packaged up this framework, along with some great open-ended question examples, in a conveniently downloadable eBook titled Core Insights: The Essential Guide to Analyzing Open-Ended Questions.

Canvs_Logo_Black_RGB-01

See Canvs in action

Schedule a demo to see how Canvs automates text analysis, saves time, and improves insights from open-ended text!

Canvs_Logo_Black_RGB-01

Schedule a demo

See how Canvs automates text analysis, saves time, and improves insights from open-ended text!

Canvs_Logo_Black_RGB-01

Almost there...

Please complete the information below to access this content.

Canvs_Logo_Black_RGB-01

See Canvs in action

Schedule a demo to see how Canvs automates text analysis, saves time, and improves insights from open-ended text!