In our experience, market research and consumer insights professionals often have a love/hate relationship with open-ended questions. From one perspective, they help uncover valuable insights, but on the other hand, they’re notoriously difficult to analyze due to the complexity and variety of human expression. In this article, we’ll clarify the difference between open-ended and closed-ended questions, how to structure and analyze open-ends to get more out of them and why emotion is key to accessing powerful insight. In short, we want to help you unlock the power of open-ends.
What are open-ended questions (vs. closed-ended)?
Put simply, open-ended questions allow respondents to answer a survey question in their own words and without predetermined choices. Typically, and if well-constructed, they require more than a “yes” or “no” response.
Here are two examples of open-ended questions:
“Describe your favorite summer vacation.”
“How did you feel about your most recent summer getaway?”
Open-ended questions such as these allow researchers to capture ideas, thoughts, feelings and emotions without restricting or biasing the respondent with a predefined set of options. You capture the respondent’s own voice. This is powerful because it gives you the ability to truly understand how an individual views the topic. In research terms, you’re capturing qualitative data, contextual insights and emotion-led data.
In the context of market research surveys, closed-ended questions are those with a predefined set of choices. This could be a yes or no question (often screeners), such as:
“Have you traveled by plane in the last 12 months?” This closed-ended question is simply looking for a “yes” or “no” response.
They may also be more complex, such as: “Which of the following means of transportation have you taken in the last 12 months?”
A) Plane
B) Train
C) Car
D) Mule
Closed-ended questions often make life simpler because the data is quantitative. They’re also ideal when a predefined set of choices are being tested for (such as the example above, when asking about a mode of transportation).
Surveys are typically testing for assumptions and built from known baselines and will be composed primarily of closed-ended questions. While a smaller set of questions in a survey, open-ended questions can reveal unexpected topics and nuanced emotional reactions.
How to craft your open-ended questions
When considering how to construct open-ended questions, you should start with words like:
What, why, how, describe, tell us about.
These provide a foundation to ensure that you’re not closing off a question or “forcing” a single-word response (though that can still happen). Our advice here is to bring real curiosity to your audience and always sense check your questions with someone to ensure you ask non-biased questions. You must avoid leading respondents down a particular path. This means being mindful of the questions you ask.
Avoid asking questions that might make respondents/customers answer defensively or even avoid the question entirely. The language you use is key. Another pitfall is accidentally constructing a closed-ended question. Consider the two open-ended questions below:
“Do you feel positive about our customer service?” and “How do you feel about our customer service?”.
Both are technically open-ended questions. Unfortunately, the first question can be answered with a simple “yes” or “no” response, so it’s not likely to yield valuable information about why the respondent answered the way they did. Put another way, the first question could just as well be structured as a closed-end (and more likely as a scale). In contrast, the second question is more likely to elicit a response such as “generally the experience was great, but I was annoyed that the lines were so long.”
This response provides depth, describes a negative and positive emotion, and provides the business with specific, actionable insight regarding why the consumer feels the way they do. Speaking of emotion, while not every open end is designed to elicit emotion, when they do, the results can yield powerful insight for the business.
Turn emotion into your competitive advantage
Increasingly, brands are seeking an emotional connection with buyers and place value on consumer meaningful empathy. Responses to certain open-ended questions are critical to understanding how consumers feel about a topic, experience or brand. To unlock emotion, you must frame your open-ended questions to uncover emotion and apply the correct methodology for analysis.
The most common question we hear is: “How should I structure an open-ended question to tease out the highest level of emotional reaction?”
Here are some examples to get you moving in the right direction:
“Remembering back to when you purchased “x”, how did you feel walking out of the store?”
“What were you hoping for when you bought this item….”
“What are your feelings about “x”
Each question is taking a slightly different approach, but they’re all phrased in a way to help capture the feelings of the respondent. Remember that any questions that you’ve previously asked may also present opportunities for you to now dive deeper and focus on emotion-led questions. The next step is to ensure that you apply the right methodology to analyze your open-ends.
How to analyze open-ended text responses
Analysis of open-ended text responses is challenging on multiple levels. First, as we’ve touched on here, open-ended questions can be used to elicit a variety of information, from an unaided response to an emotional feeling. Second, real people don’t answer in consistent, grammatically correct sentence structure. People use slang, sarcasm and even emojis (less in surveys, but certainly in places like product review ¯\_(ツ)_/¯ ).
With this in mind, there are several important elements for analysis to consider:
Topics: This is performing a general content analysis of what people are “talking about” in the open ends. The key here is distilling or classifying natural expressions into topics and/or themes.
Codes/Nets: This is the hierarchical categorization/classification of themes as expressed in the open ends utilizing a code frame.
Sentiment/Emotion Analysis: This is extracting general sentiment and/or nuanced emotional reaction from the text. Slang, emojis and idioms can make this more challenging!
Understand the whole picture
Given the potential volume and variety of open-ended text (especially when we consider use cases like customer feedback surveys), insights professionals have historically found analyzing open ends to be challenging simply from a bandwidth perspective. But, if we can’t analyze all of the responses then we risk failing to capture the bigger picture; we risk analytical confidence. Unfortunately, the most common approaches to the analysis of open-ended response come with trade-offs:
Manual coding: Hand coding of open-end is time-consuming (and/or expensive if outsourced), tedious and prone to human error and bias. Large data sets magnify this issue, so many organizations will code just a sample of the open ends from larger surveys.
Basic automation: Tools like excel provide basic, word-level automation (word frequency, word clouds, etc) and some will also provide basic sentiment analysis. However, these tools are typically blunt instruments, unable to understand complex expressions, like sarcasm and memes, rationalize misspellings or concatenate similar ideas into a single category.
Selective quoting: Without the tools to systematically process open-ended responses, many organizations will simply use verbatim quotes to support or illuminate the qualitative findings of the study.
New research technology like Canvs AI is giving insights professionals the power to automate or semi-automate the coding of open-ended responses, bringing quantitative scale and statistical significance to this form of qualitative feedback. This has the potential not only to accelerate discovery and enhance analytical confidence but also to expand the potential for open-ended questions in research.
This revelation should be heart-warming to insight professionals.
We hope this article has given you food for thought about how to truly get the most out of your open-ends, if you’d like to learn more about Canvs then simply click here.
