Don’t (just) ignore user feedback

Involving users early on in software development processes is absolutely essential. This insight has now become widely recognized. Unfortunately, however, simply saying, “We’ll just ask our users,” isn’t enough. People often rush into the process with half-baked methods, and the user feedback then ends up, without any reflection, directly on to-do lists or sprint boards in the form of proposed solutions. This can have serious negative consequences for the product. So how can feedback be collected in a structured way, and what does that tell us about the people who interact with the product? How do we ultimately learn to build the right product without blindly following the wishes expressed? Here are a few suggestions.

The Homer, or how not to do it

A perfect example of blind obedience to user requests, and the resulting outcome in product design, can be clearly seen in “The Simpsons”: Season 2, Episode 15, “Oh Brother, Where Art Thou?”. Homer, as a representative of the typical average American, is asked to design his dream car. What follows? Clear requests for features: 

• Two glass domes: one for the driver, the other for the children, complete with restraints and muzzles for them
• An apocalyptic engine noise
• Three horns, all playing “La Cucaracha”
• An oversized cup holder

Spoiler alert: The car turns out to be a flop. Homer alone would probably have been happy with it, but even that’s not certain in the end. After all, you never see him driving it.

A lot of feedback leads to a better understanding

The story sounds exaggerated, but it happens time and again in software projects in a similar form: Users, often not the quietest ones, are chosen to represent an entire group and asked about their wishes, which are then fulfilled one-to-one. Or even worse: Many individual voices speak up, each with their own opinion, expressing personal needs, or simply describing desired features that they hope will enhance their product experience. Product managers who lack a clear stance and vision will easily succumb to the noise of these feedback voices, true to the saying: “If you don’t stand for anything, you’ll fall for everything.”

So does that mean we shouldn’t listen to our users? Far from it! Lots of feedback is good and important. The key point is: What do we do with it? Feedback isn’t a to-do list that gets added to the backlog; it’s data points on the path to a better understanding of our user groups. But only by properly synthesizing all these data points can we draw helpful conclusions about product design. Users themselves shouldn’t be directly involved in designing the product. They have a problem that we’re supposed to solve. And that’s the level of the problem to which we want to transform the feedback. 

Saying vs. doing

Jakob Nielsen provocatively wrote in a 2001 article, “First Rule of Usability? Don’t Listen to Users.” What he means is that people are very bad at describing their own behavior: past, present, and especially future behavior. You can’t trust a statement like “I’ll definitely use this feature often,” even if it’s said candidly and honestly. 

It’s easier to observe whether goals are being met than to describe it

Nielsen essentially suggests observing people as they complete their tasks to achieve an overarching goal. If the goal is, for example, “attending an opera in Munich,” the main task is “buying a ticket online,” from which subtasks such as “selecting a date,” “selecting a seat,” or “paying” are derived. Contextual analyses or usability tests are excellent ways to observe how easy or difficult these tasks are to complete at any given moment. It’s hard to lie in this context: The solution either works well, with some difficulty, or, in the worst case, not at all, but it’s hard to sugarcoat an unfulfilled task. Incidentally, the subjectively perceived user experience can be very good, even though the test participant may have just purchased a ticket for the wrong performance without realizing it. She is happy but does not (yet) realize that the task has not been completed. Here, too, what actually happened carries more weight than the emotion expressed.

Be aware of distorting influences

Especially when it comes to explicitly verbal feedback, the context in which it was expressed is of immense importance. Often, unconscious, biased attitudes or distortions, also known as bias, creep in, which strongly influence judgment and thus the truthfulness of the statements. Comments from focus groups, for example, are often heavily influenced by the prevailing group dynamics and implicit expectations. If four out of five participants in a focus group express clearly negative opinions about a product, it depends heavily on the personality of the fifth participant whether he or she might still choose to voice a potentially positive experience. If in doubt, this positive experience goes unmentioned, and the group’s overall assessment ends up being 100 percent negative. Ultimately, a single dominant voice can be decisive in significantly influencing the tone and, consequently, the overall picture. In individual interviews, the result would have been a very different, more nuanced assessment. You must be aware of these factors when using data collected in this manner.

Excessive bias renders study results ineffective

Another example from the field of usability testing aptly illustrates the “say vs. do” effect under the influence of bias. Imagine you’re selecting the test participants yourself for the planned usability test of your company’s online store. To do this, you choose only premium customers you know personally, whom you incentivize in advance with a very valuable gift basket that includes a personal thank-you note. During the testing, you can observe the participants as they fail to complete the tasks assigned to them. For example, they might not be able to find a specific product using the search function, yet at the same time describe the user experience as consistently positive and praise you to the skies. It’s like they’re saying, “I can’t find the sneakers, but it’s so beautiful here.” In this case as well, the observed behavior is the more relevant basis for product improvements, while the explicitly stated experience stands in stark contrast to it and, as expected, should have been marked by much more frustration. The desire for social approval is definitely at work here, also known as social desirability bias, and there is certainly also a sampling bias, or selection bias. As a selected premium participant, no one wants to come across as ungrateful; the champagne from the gift basket was simply too delicious for that. However, these factors are no longer minor distortions; rather, they call into question the validity of the entire research project. In this case, the solution is to recruit new participants independently, or to terminate the study.

If you have a vision, you should go to the user

In addition to collecting data with as few biases as possible, the Double Diamond model from Design Thinking offers a better approach to handling feedback: listen, ask questions, and seek to understand before moving on to implementation.

Problem area: understanding

During the “Understanding” phase, a time of exploration, the goal is to generate as much feedback as possible, and as much as is meaningful, from as many available channels as possible within the problem space. This can include quantitative data from a web analytics integration, results from online surveys, or KPIs such as the retention rate, all of which help paint a vivid picture of user behavior. For products that are already in the hands of users, it can be helpful to integrate feedback tools directly into the application or to conduct regular usability tests to identify current pain points and thereby supplement the quantitative data with qualitative insights. The goal is to move from asking “What’s happening?” to “Why is it happening?”

However, at this stage, there’s usually too little information to blindly trust the feedback and proceed directly to implementation. For example, if a user says, “I’d like an Excel export for this table,” that would be a clear indicator to seek out end users who are expressing exactly this need. Now it’s time to keep asking “Why?” until you’ve gotten to the heart of the problem: “What’s supposed to happen with the data in Excel? Why and how will the data be reorganized in Excel? Will the data be transferred from Excel to another tool? How has the data been handled in the past? Do your colleagues have similar needs?” And so on. 

Problem domain: Define

In qualitative research, the “Why?” question can be used until a sense of saturation with the original questions sets in and a sense of the underlying context emerges. Once a clearer picture of the issue begins to emerge, the next step is to consolidate the extensive data collected during the convergent “Defining” phase and translate it into a clear problem definition. 

This definition can lead to artifacts such as personas or scenarios, which clearly explain where a specific user group’s need for the mentioned Excel exports comes from, what pain points this need is intended to alleviate in their daily work, and what specific tasks and goals are associated with it. At best, data-driven persona modeling can also help you properly contextualize individual feedback in the future. Is a request for a specific feature an isolated, unusual request, or does it address a known pain point of an archetypal user group that you definitely want to cater to? You’ll now be better equipped to answer these questions.

A good vision provides focus within the solution space

When applied to the product as a whole, the problem definition can also be formulated positively as a vision. This serves as a guiding star for the project, setting the direction without specifying exactly how the product will solve each individual problem. Once a team has internally clarified for whom and in what context a particular problem is to be solved, it becomes easier to maintain a balanced approach to feedback from all directions. And ideally, there’s still enough openness to revisit the problem space when recurring feedback patterns emerge, to see if anything has been overlooked or perhaps not yet fully understood. Understanding users and their problems and needs remains an ongoing learning process, but it is essentially something completely different from fulfilling users’ wishes on a one-to-one basis. Only with a clear vision of your own actions and the value you aim to create for end users will you no longer be helplessly at the mercy of external feedback and be able to systematically integrate these influences into your own strategy. 

An anecdote to wrap things up

The phone support department of an online store is tasked with working more efficiently during customer service calls. When you ask and observe the staff members in that department during their daily work, they complain that end customers, when trying to identify themselves over the phone, either provide their store account number, an order number, or, by mistake, the shipping carrier’s tracking number. All three numbers are spread across different fields on the service representatives’ screens, requiring them to switch back and forth between them, which is cumbersome during a phone call. As a result, the first few minutes of the call are tedious for everyone involved because they first have to “find” the right information. An articulate employee from the call center suggests: “Let’s put all three input fields on one page, then we won’t have to scroll anymore and will have everything at a glance.” The development team gratefully accepts the feedback; the solution sounds sensible and comes from a daily user. So, in the existing system, the three input fields are consolidated onto a single page, and the team hopes this will lead to increased efficiency.

If we were to go back to the beginning and revisit the question of “why,” the problem description would focus less on input fields and more on the service department’s task of identifying customers as efficiently as possible in order to discuss a current issue with a recent order. So, if you look at the problem from a broader perspective and don’t close yourself off to alternative approaches in the solution space, by using appropriate ideation methods, you might come up with ideas, for example, on how to identify callers using the service department’s already reasonably smart phone system and display the current order on the screen before service representatives have even picked up the phone. However, the employee providing the feedback hadn’t given such automation any further thought because the initial identification had become an integral part of his daily work routine, and the phone system had not, to date, offered him many opportunities for interaction.

Good UX goes beyond current system limitations

Users who work in a system every day for years often think within the confines of that system when they provide feedback. Yet it’s our job as developers and product designers to look beyond these limits and, within the scope of current technical capabilities, build applications that provide lasting solutions to our users’ fundamental problems. This is only possible through a deep understanding. And although even individual pieces of feedback can contain a gem of innovation, we should retain the final say on how problems can be most effectively solved for end users. 

It’s a bit like going to the dentist. The dentist wants to know where it hurts, for example, the upper right side. He doesn’t want to hear that you want a root canal followed by a porcelain crown on tooth number 17. As a rule, the dentist will first take an X-ray himself to see what’s going on. We should follow this example in a figurative sense as well.