Analyzing open-ended survey questions: the “Essential Guide” Framework

The twists and turns of today’s dynamic consumer landscape means it has never been more important to have your finger on the pulse of evolving consumer preferences and behaviors, and what’s driving them. Insights from closed-ended questions tell you a lot about the consumer, but often missing is a deeper narrative around “why.” This is where analysis of open-ended survey questions comes in, allowing you to start with a blank canvas (no pun intended) that gets painted by unfiltered responses relatively free of question bias. The answers to these questions help you better understand consumer preferences, attitudes, and feelings, as well as identify additional opportunities for further in-depth analysis. This post will present a simple framework for analyzing open-ended survey questions. 

Through our work helping hundreds of organizations get more value from open-ended text, we’ve developed an analysis framework for open-ended survey questions. This framework can be applied across different types of research based on the objectives and structure of the particular study as outlined in our eBook Core Insights: The Essential Guide to Analyzing Open-Ended Questions.

Framework for analyzing open-ended questions

To generate core insights, analysis of open-ended questions needs to go beyond simply counting the words and phrases, or indicating binary sentiment (i.e. positive/negative). Researchers want to know what the respondent means, and, if possible, why. A combination of three related lenses of analysis can be used to unpack the meaning and narrative of open-ended text: thematic analysis, emotion measurement, and correlation. 

Thematic analysis of unstructured text

Thematic text analysis is the process of parsing the ideas or things that are expressed in text and presenting those ideas in a structured way. The kinds of questions we’d approach with thematic analysis will be the ones where respondents are answering “what” questions, such as “what was the main idea in this episode?” Or “what brands come to mind when you think of shaving cream?” Thematic analysis is a broader version of topic analysis. Asking the respondent to describe their “perfect ice cream treat” might include responses like “Tillamook,” “Ben & Jerry’s,” “mint chocolate chip,” “local,” or “waffle cone.” These could be considered and scored individually as topics, but likely take on more meaning when organized into broader, hierarchical categories, like “brands,” “flavors,” “serving types,” and “other qualities.” Many researchers will recognize this hierarchical approach as equivalent to a code frame. While basic topic analysis may suffice in some studies, like unaided brand questions, this second level of thematic analysis generates additional, potentially more nuanced insights. 

Emotion analysis (aka Sentiment Analysis)

Emotion analysis is an extrapolation of how the respondent “feels” based on how they have expressed themselves. This is a more nuanced version of sentiment analysis, which typically categorized statements into positive, negative and neutral (more on this below). Generally “how” or “why” questions are more likely to generate an emotional reaction vs. “what” questions. Examples include questions like  “How do you feel about the last season of the show?” and “Why did you give that rating?” Naturally, some topics are inherently more likely to generate emotional reactions vs. others. For example a bank asking “What did you like best about your experience?” is likely to register a different level of emotion vs. an amusement park asking the same question. 

There are several popular emotion measurement frameworks that can be applied to open-ended text. The most basic of course is a binary sentiment analysis of positive and negative (plus neutral). While the simplicity of this can be helpful, modern language, what Canvs calls the “unnatural language” of social media, makes discerning positive and negative challenging (particularly for AI). “The brand is dope” might be interpreted as positive, while “Politicians are dopes.” might be seen as negative. We prefer a judgment-neutral framework that identifies discreet emotions. The Eckman framework utilized 7 basic emotions of anger, contempt, disgust, enjoyment, fear, sadness, and surprise, while the Plutchik model identifies eight primary emotions: anger, anticipation, joy, trust, fear, surprise, sadness and disgust. Canvs has developed a proprietary, 42-emotion model for emotion measurement to provide additional nuance for researchers, but also allows researchers to switch frameworks and compare between them. Whether conducted by a person or an AI, the “art” of emotion measurements is in the interpretation of language and applying that interpretation in a consistent, unbiased way. 

Correlation analysis of text

Correlation analysis is the ability to identify statistically relevant connections between information from the open-ended text and associated data from the close-ended questions, such as respondent demographic cross-tabs. For example, you may find that a specific age range prefers one type of nut butter vs. another, respondents with different income levels have different preferences in their “perfect ice cream treat,” or that women describe an ad creative with different emotional words vs. men. You could also find connections between different attributes of a product described in the open-ended text and the emotional reaction (i.e. correlation between two or more data from the open ends themselves). 

Applying These Lenses to Market Research

These analytical approaches to open-ended questions can be applied slightly differently across various types of research, largely driven by the topic and objectives of the project.