Consumer expectations, preferences, and behaviors are evolving at an accelerated rate, so it’s never been more important for brands to take a human-centered approach to their product development, understanding the problems their users experience every day, and how they experience their offerings or services. Product user testing a critical tool for consumer insights teams to understand consumer needs and preferences. User testing takes many forms, from central location testing (CLT), focus groups and in-home user testing (iHUT), to feedback from customer communities in the form of consumer journals and diaries. 

It’s critical for product research to get as close to the consumer as possible (and to get the details right). One key but often overlooked resource in this pursuit is the analysis of open-ended questions from feedback surveys, sensory lab ballots, consumer diaries and iHUT transcripts, and product test questionnaires. These provide a unique opportunity for unfiltered feedback from the consumer filled with detail and nuance, but are often challenging to analyze with scale and consistency. Insights teams must generally hand-“code” (i.e. manually analyze) open ends (which is tedious and time-consuming), utilize surface-level language processing (such as word clouds) or cherry-pick from the verbatim text to support the qualitative findings.

Brands that wish to keep pace with consumers must take advantage of every touchpoint they have with their users. When users share their feedback via verbatim text it’s essential to analyze and summarize the responses quickly, identifying major themes and segments that can be utilized across the product development lifecycle.

Expect the unexpected, like corn in the mashed potatoes

The value of open-ended questions in product research is to capture the unaided, unfiltered feedback of the consumer. While it’s likely that you may be “testing for echo” around well-known product attributes using tried-and-true survey methodologies, open-ended questions provide a confirmatory approach of identifying the most prevalent attributes, as well as the emotions evoked by the product.

Open ends also provide a source for unexpected insights or feedback that might not be possible to capture with closed-ended questions, such as scales and purchase intent filters. This includes detecting and pinpointing quality issues with products or experiences. Consider this scenario recently shared by a customer: A researcher is given the task of understanding the performance issues of a frozen dinner product. The product receives poor scores in the survey’s close-ends, but it is difficult to identify the reason why… until the open ends are analyzed. Here themes like “messy” “smooshed” “corn” and “mashed potatoes” appear, leading us to discover that the packaging is causing corn to get into the mashed potatoes! 

How are product feedback open-ends analyzed?

The approach to analyzing open-ended questions varies based on the type of question asked and the goal of the research, but often rely on both emotional and thematic analysis. 

Emotional analysis allows researchers to identify the varying levels of enjoyment, anger, and fear users can feel when experiencing their products. For example, consider a question like “what were your initial thoughts or feelings after trying the product?” A researcher might identify that users are “excited” about a resealable lid vs “love” the richness of flavor. Or, they might identify which users are just slightly “annoyed” about a product’s new shape, vs feeling “rage” about an increase in prices. Emotional analysis can also be valuable for comparisons – to identify which new flavors bring feelings of “nostalgia” or which labels on new packaging are “confusing” or leave users “unsure.” 

Questions about specific preferences or features like “why did you choose sample XX as your favorite?” or “what would you change about the product?” often elicit nuanced responses. For these questions in particular, it’s important to understand what themes and learnings are important for your stakeholders and research goals. For example, analysis of a food & beverage feedback study is likely to focus on specific attributes like the taste/flavor (spicy, too sweet), consistency/texture (rubbery, creamy), smell (overwhelming, nostalgic), mouthfeel (sticks to my teeth), appearance (too neon), etc, while analysis of a consumer electronics feedback study might focus on themes like usability (difficult to use, user friendly), pricing (affordable, expensive), reliability (tech issues, trustworthy), etc. 

Understanding and disseminating context about important themes is vital when analyzing open ends in product user testing. For example, the word ‘salty’ might refer to conflicting themes of “love the salt” vs “too salty” on the single study in the food & beverage industry. However, in other industries, “salty” has a totally different meaning. For example, on a beauty survey if someone says “I’m just salty because the price was raised,” the theme “annoyed about prices” has nothing to do with the ingredient “salt.” 

Once one has identified the key emotions and themes found in open-ended product feedback, it’s time to discover the story those themes tell. Cross-tabulation analysis (aka cross-tabs) is often a starting point for uncovering trends and differentiation between various cohorts or products within your survey sample. This analysis is likely to start with filters around overall liking and/or purchase intent. It could include demographic segments such as male/female, geographic region, age, etc, against responses to a particular closed-ended question (for example, how recently they purchased a particular product, or how often they use it). Researchers can confirm assumptions found in close-end responses, identify key opportunities to bring to the product development team, and build strategies for segmentation.  

Get to the heart of consumer reaction

Open-ended feedback can be gold for brands looking to understand and respond to consumer feedback on their products, but they are often ignored due to the manual effort required to summarize themes and communicate a useful story with the data.

To take advantage of the existing opportunity your open ends represent, and respond to consumers with agility, researchers need the ability to automate (or semi-automate) the summarization of open-ended responses, bringing quantitative scale, speed, and statistical significance to this form of qualitative feedback without losing the ability to interpret nuance and emotion, generating actionable insights that feed the product lifecycle.