Continuous Customer Feedback Programs, Part 3: After the Sessions

August 19, 2013

In Part 1 of this three-part series, we described how to set up a continuous customer feedback program to obtain the qualitative data that your product team needs. Part 2 covered conducting customer feedback sessions. Now, in Part 3, we’ll discuss post-session activities, as well as the outcomes of conducting a continuous customer feedback program like ours at IBM.

Post-session activities are perhaps the most time-consuming and important aspect of running a successful customer feedback program. Our experience has proven that neglecting to process and act on the information that your customer sessions reveal is likely to result in your customers’ feeling frustrated and wondering why they spent their valuable time providing their feedback—only for it to be ignored. Therefore, it’s essential that you spend time processing customer feedback and correctly route it back to the development team and other stakeholders to enable the eventual resolution of your customers’ issues.

Champion Advertisement
Continue Reading…

You should conduct post-session activities with one major goal in mind: transforming your customers’ feedback into actionable design or development deliverables. In fact, this should the main goal of a customer feedback program in the first place. The creation of actionable design or development deliverables that are driven by customer needs provides both tangible and intangible benefits to the customers who participated in your customer feedback sessions, to your development team, and to you. In this article, we’ll describe our best practices for conducting post-session activities.

Processing and Organizing Your Data

Now that you’ve gotten all this feedback, what should you do with it? Consider following these tips when you’re processing and organizing your data.

Disseminate Your Notes from the Sessions and Get Clarification Where Necessary

You might have used various ways of recording the issues that customers discussed during their sessions. We’ve found that a combination of recording the sessions and taking detailed notes is the best solution for us. (You must obtain permission from participants to record the sessions. If certain customers do not agree to being recorded, ask them to leave the session.) Recording the sessions offers a few benefits. Recordings are useful for gathering historical data. You can pass the recordings along to someone who was unable to sit in on a call, but is interested in the discussion. You can go back to the recordings for clarifications on your notes, if necessary. We tend to take very detailed notes during the sessions—bordering on transcription—to maintain a sense of the discussion’s context. We find this type of notetaking more valuable than simply capturing bullet points—if only to ensure that we’ve captured exactly what participants have said, without introducing any unintended bias.

Review your notes to determine what issues your customers are having, noting their severity and frequency. Is an issue preventing task completion and goal achievement? Have multiple participants weighed in with their agreement on an issue? Decide whether issues are indicative of defects in your product or the need for enhancements. Consider framing the issues as user stories to communicate the user experience aspects of the issues.

Keep the Larger Themes in Mind

As you continue processing and tracking individual issues from customer discussions, they'll inevitably become unwieldy and difficult to manage. One way of mitigating this problem is to keep an eye out for emerging themes. It is much easier to track issues under the umbrellas of particular areas of the user interface rather than working with them as individual issues. For example, if you have three issues relating to a shopping-cart widget, consider grouping them in a checkout bucket. This lets you unify related issues and present a more cohesive picture of customers’ problems to development stakeholders. By linking related issues, you'll be able to present a more accurate picture of the improvements that need to take place within the context of a workflow rather than describing individual items that can seem innocuous on their own. Whenever possible, add supporting information such as task frequency to your themes.

Input Customer Issues into a Change Management System

Since our organization has an established change management system and processes in place, we simply enter all customer-identified issues into the system as we uncover them. We can communicate with customers through the change management system, enabling us to use it to obtain clarifications or solicit further details. All of this information goes directly into the system where developers pick up changes for implementation. This change management tool facilitates direct communication between User Experience, customers, and Development, enabling us to discuss issues and determine their resolutions—all within our open, commercial software development model.

Communicate the Issues to Your Customers

Processing and organizing the data from customer feedback sessions is time consuming. Don't forget to highlight your effort by communicating the issues back to the customers who generated the data. If you are using a collaborative change management system, invite your customers to add further feedback to the issues. If not, consider sharing the individual issues during regular checkpoint meetings and soliciting further comments or discussion. The purpose of communicating issues back to customers is twofold:

  1. Customers will know that you’ve heard them.
  2. You can gather more information on issues, including validation of issue severity and priority.

While this work can be tedious, the time you spend organizing and processing customer feedback is time well spent. Of course, there are other ways of compiling customer feedback. Regardless, the main purpose of processing customer feedback is to transform it into easily consumable information that is persuasive to your stakeholders.

Communicating Customer Feedback

You’ve organized and processed your data. Now, determine who should hear your customers’ feedback and can do something with it.

  • Find the decision makers. Who is empowered to decide what goes into your product? If you aren’t already working with these people, locate them and introduce yourself. Even better, get their buy-in and support for your user research. Ideally, you did this before you even started!
  • Decide how to review the feedback with these decision makers. Determine how to communicate your findings in an effective way. Would your stakeholders appreciate receiving regular summaries, with supporting conference calls? Or are they knowledgeable about the issues and able to rely on the change management system alone? In what format would this data be most easily consumed in your organization and at what level of detail? Think about creating a framework to support your research and let you easily distill metrics and priorities from your data.
  • Enable stakeholders to hear your customers’ feedback in real time by inviting them to participate in the sessions when appropriate. Hearing the participants’ thoughts first hand, they may take their own to-do’s from the sessions. Integrating team members into the sessions can also minimize the work that you need to do to summarize your research findings, because stakeholders are already familiar with the context and the issues.
  • Don’t forget to reflect your customers’ feedback back to them. This lets you verify that you have understood their issues correctly and get more details, if necessary. Be sure to communicate that you’ve heard their feedback and are acting upon it. For bonus points, maintain metrics as you go. Keep a running count of the issues and themes that you’ve generated, and track how many issues have gotten resolved. How many enhancements have resulted from customers’ feedback? How many have been implemented? How many are in planning? Not only does this make participants feel gratified, it can help you to communicate the value of your research to your management team. Lastly, if you don’t own the outcomes of your findings—for example, you can’t deliver code to address them—be sure that your customers know that. In this case, clearly communicate that your role is to facilitate user experience research and pass the results along to development.

Making your stakeholders aware of your research and establishing frameworks and expectations around the resulting customer feedback helps to prepare them to make use of the results. Communicating your findings in a way that stakeholders can easily consume often prevents the need for more rework in the future and may help them to address your feedback sooner than if you had communicated it in an ad-hoc manner. Finally, communicating your findings back to the customers who participated in your research is imperative to validate your own understanding of the issues and make them feel that the time they spent providing input was time well spent. Providing metrics at any given point is especially helpful in this light.

Establishing Priorities for Development

Make it easy for development to implement your change requests by prioritizing the issues.

  • If you’re keeping an eye out for emerging themes, consider prioritizing them as you go. You can use whatever priority scale works best for you and your stakeholders. We typically start by identifying frequently occurring issues versus severe issues. Look for areas where multiple customers across various industries have given you the same feedback. Don’t forget that seemingly innocuous issues can add up to a severe usability problem when they occur in the same context.
  • Determine priorities from your customers’ perspectives. Use a prioritization tool to put together a prioritized list of customer issues. Remember that, during this exercise, you may be talking to members of various target audiences that have different priorities. For instance, an administrator would rank items differently from a practitioner.
  • Triage the issues and review your work with stakeholders. Be ready to speak to each issue in depth, communicating both the issue and its resolution. Your goal is to clarify the requested work so you can hand it off to development.

Your research will be more actionable if you can communicate the priority levels of the work that you’re requesting. Without this, you risk developers’ seeing all of the feedback that you’ve gathered from customers as low-priority, nice-to-have issues.

Benefits of Conducting a Customer Feedback Program

Our continuous customer feedback program clearly provided tangible benefits in terms of product improvement. In one series of recurring meetings that took place over a period of four months, the group generated over 120 enhancements. Of these, we addressed more than 70 enhancements in the product’s next major release. During this time, the program’s intangible benefits also became clear. Customers were more likely to participate in usability evaluations and other collaborative endeavors after having participated in the feedback group—likely because we had established a personal relationship with them, as well as because of their knowing that we considered their feedback valuable, took action on it, and addressed their issues.

Another unexpected benefit of the sessions was that they offered a forum for users to talk to each other about their experiences using the product. Participants ranged from subject-matter experts to new practitioners, and they were able to discuss various features and usage patterns of the product with other users, in production environments for real deployments. Some participants expressed their satisfaction with this aspect of the calls several times, and although the meetings had a planned end date, they asked for the calls to continue because they were getting such immediate value from them.

If you decide to try conducting your own customer feedback program, be prepared to do a lot of work and expend a lot of time, but you’ll also gain many benefits. Just remember to engage your stakeholders, provide value to the participants, remain unbiased, and make it easy for your development team to act on the findings that result from all of your hard work.

Encouraging Continual Customer Feedback

Over time, you’ll hopefully evolve and maintain an effective method of obtaining rich, continuous customer feedback. Your product development team will come to rely on your bringing customer feedback to the team and use your research to make informed design and feature decisions. Great!

What else is there to do? If you want to further encourage continual customer feedback and design collaboration, here are a couple of approaches to try.

You’d be surprised at how much an in-person meeting can energize participants. Are there any conferences they’re likely to attend that you could attend as well? Local user group meetings? Are any of your participants local and available for an in-person visit or contextual inquiry? While limited travel funds may typically require you to obtain most of your customer feedback remotely, search for opportunities to meet customers in person. Is anyone on your team traveling to a customer site? Are there any local sales contacts who can put you in touch with their customer accounts?

Along with exploring any opportunity for in-person research, you can continue to use your customer group to generate continuous excitement and motivate them to spend more time providing feedback. While this requires going a bit above and beyond your regular sessions, it significantly improves your customer relationships, as well as their perception of the value of your work. Offer one-on-one sessions where appropriate—such as a deep dive on a particular feature or workflow or a conversation with an interested and willing developer. Whenever possible, frame one-on-one sessions as an opportunity to preview in-progress designs and influence the final user interface. Offering one-on-one usability testing sessions of incomplete implementations—remotely, if necessary—offers your customers a chance to try a new feature for themselves. You might be surprised by how enthusiastically your participants might market your research for you!

Needless to say, if you start conducting customer feedback sessions outside your regular group, it’s still important to transform the feedback into actionable issues and communicate them back to both the participants and your product development team. Obviously, additional customer communications, touchpoints, and research activities require additional work. In our experience, one-on-one conversations—whether remote or in person—can foster a more amicable relationship with your customers, encouraging them to reach out to you when they find usability issues and enabling them to freely and honestly share their feedback during the regular group meetings. 

Senior UX Researcher at Nokia

Ottawa, Ontario, Canada

Marnie AndrewsAfter graduating from the University of Waterloo’s co-op Computer Science program in 2000, Marnie spent her first 8 years working in the software industry splitting her time between programming, interaction design, and usability at small companies that valued her ability to move between skills. Then in 2008, Marnie became a full-time UX Designer at IBM. With her passion for designing with real users, she has led cross-disciplinary product teams to improve the quality of their deliverables by leveraging users’ feedback. Now, as a Design Manager, Marnie ensures that the right people are on the right projects. Her team creates design solutions using the best methods possible. Marnie is currently creating a design research program with her team at IBM Business Analytics.  Read More

User Researcher at IBM

Boston, Massachusetts, USA

April L. de VriesAt IBM, April manages usability activities and deliverables for an enterprise quality-management product. Her day-to-day work includes conducting usability research, creating user-centered design deliverables, engaging in extensive cross-functional collaboration, and transforming client feedback into actionable UX design and development work. April’s academic background includes a BS in Information Design and Corporate Communication and an MS in Human Factors in Information Design, both from Bentley University near Boston.  Read More

Other Articles on User Research

New on UXmatters