In this edition of Ask UXmatters, our panel of UX experts discusses how to explain the need for user research and usability testing to a client who wonders why an expert review is not enough.
I hope you enjoy this month’s lively discussion about the best ways to persuade a client that user research and usability testing are a necessary part of a project. Why is it essential for UX designers to rely on user research and usability testing? What value does getting the views of users provide? What data already exists about users and what are the gaps in that knowledge? How can all of this data support your UX design work and deliver project outcomes that deliver value to your client? You need to convey the answers to all of these questions to your clients—and be sure to connect your answers to your clients’ business reasons for starting a project in the first place.
In my monthly column Ask UXmatters, our experts provide answers to our readers’ questions about a broad range of user experience matters. To get answers to your own questions about UX strategy, design, user research, or any other topic of interest to UX professionals in an upcoming edition of Ask UXmatters, please send your questions to: [email protected].
The following experts have contributed answers to this edition of Ask UXmatters:
Carol Barnum—Director of User Research and Founding Partner at UX Firm; author of Usability Testing Essentials: Ready, Set … Test!
Leo Frishberg—Product Design Manager at Intel Corporation
Steven Hoober—Mobile Interaction Designer and Owner at 4ourth Mobile; author of Designing Mobile Interfaces; UXmatters columnist
Peter Hornsby—Web Design and UX Manager at Royal London; UXmatters columnist
Jordan Julien—Independent Experience Strategy Consultant
Tobias Komischke—Director of User Experience at Infragistics
Cory Lebson—Principal UX Consultant at Lebsontech; President, User Experience Professionals’ Association (UXPA)
Baruch Sachs—Senior Director, User Experience at Pegasystems; UXmatters columnist
Daniel Szuc—Principal and Cofounder of Apogee Usability Asia Ltd.
Jo Wong—Principal and Cofounder of Apogee Usability Asia Ltd.
Q: Has a potential or an existing client ever said to you, “You’re the expert, so why do we need to do user research or usability testing?” What is the best response to this challenge from a client?—from a UXmatters reader
“I usually answer that it is because I am the expert that I absolutely need this research,” replies Baruch. “User research and usability testing are part of my expert toolkit—just like a level or the right drill bit is part of an expert craftsman’s toolkit. Be wary of any UX experts who feel that they do not need to prepare properly to create their designs or to validate them. Companies may hire experts to come in and tell them how to do things, but they’re not going to be able to do everything themselves. Think of UX experts as almost like general contractors: they know how to design, build, and implement everything, but they are master coordinators as well. They don’t usually do every construction job themselves—and, if they do, they may lose sight of the bigger solution.”
“I tend to think about getting the design right as a series of steps,” answers Peter. “First, UX designers use their knowledge and experience to get a design as right as possible, using the information that is available to them. For example, they need to understand both business and user goals and how to align them, and they must be aware of existing user research. UX designers use this information to create designs—lots of designs! Then they do usability testing to get feedback on their designs from users. Unless a team has a huge testing budget, this testing gets done in layers. First, a team might do some testing locally, using whatever people are available. Then, they’ll refine their designs. Next, once they’re comfortable with their designs—but not so comfortable that they’re unwilling to change them!—they’ll do some usability testing with representative users. At each stage, they make the best of the resources they have.
“As a UX designer, you are an expert, but your design must be informed by the environment in which you’re working—and not just user research and usability testing, but also the business context, the process you’re using, the available technology, and the available funding.”
Acknowledging the Limits of the Genius Designer
“In ‘5 Design Decision Styles. What’s Yours?’ Jared Spool outlined the best case to make here,” responds Steven. “All too often, clients hire people like us to be the genius type. We’re supposed to use our knowledge and just make a decision. And if we’re good at it, we can make a good guess. But—and I tell my clients this—it’s only a guess. If we have enough time and information, we can make pretty good guesses—basing them on cognitive psychology and physiology, not just applying best practices and trendy patterns. But each business, each product, and each customer is different. Without at least a little research, we can’t tell whether our guesses are right.
“My clients often ask me to just apply best practices. Facebook does this, so why don’t we? Well, I don’t know why not. Because I don’t know what Facebook’s goal is for a design, what they did to support their design decision, whether the solution would work only in their context, or even whether it works at all. Do you notice how they change their user interface all the time and, in general, how design trends come and go? These are hints that we don’t have perfectly solid recommendations on what best practices are—unless we follow the exploration and validation steps in our process.”
Getting an Unbiased View from Users
“The benefit of asking users—who are outside the project context—to try to complete tasks using a design solution is that users approach tasks naturally, without any particular bias,” reply Dan and Jo. “They do not have any political leanings or in-depth project knowledge that would influence the results. We gain access to users’ natural language, their life context, other bits of knowledge that they may use within the context of a task and a domain, and the personal and digital ecosystems that are at play. We may also discover a usability or design principle that addresses a user need.
“As experts who a client has brought onto a project, we may complement research data by identifying areas that we think a design needs to address. But our reports may focus too much on areas that are not important to users or, ultimately, the business. Also, by confirming what has concerned the business for some time, the voice of the user—or voice of the customer—often holds more weight with a business. So we should both do expert reviews and get user feedback. Ideally, we should blend our expert knowledge and user feedback together so we can tell a stronger story overall.”
Making the Case for Research Because We’re Not the User
“Because I’m not the user!” exclaims Carol. “Seeing is believing, and no one can show your client what works and what doesn’t, what pleases and what puzzles, what engages or bores the user better than the user. The beauty of the convenience and cost effectiveness of small usability studies is that, in a day or less, your client can learn from their users and apply that learning directly to product improvement.
“If your client doesn’t believe that you need to do research, ask them to watch just two usability sessions. During the first session, your client might have doubts about the quality of the participant. But after two sessions, in which they see the same issues cropping up, they may put their doubts aside. If you can get your client to watch three sessions, they’ll be hooked.
“If you can’t get your client to watch live usability sessions, do the next best thing and show them video highlights. A good 10-minute highlights tape can go a long way toward bringing the users’ experience to your client.
“Whenever I get asked to do an expert review, or heuristic evaluation, I always ask why the client wants an expert review rather than usability testing. If the answer is to clean up the product after the review, then test it, I know that they understand the strength of this two-pronged approach to building usability into products. But if the answer is that they don’t have the time or budget for testing with real users, push back and ask how much time and budget they have. Then show them how they can do testing within both their timeframe and their budget. If they can do only one research activity, make your best case that it should be testing with real users.”
“Since the bulk of my work is research and evaluation,” answers Cory, “I actually hear this question or a derivation of it fairly regularly. Stakeholders wonder why they shouldn’t just hire me to do a heuristic or expert review. My answer to them is that, while I can review a user interface with respect to general best practices, in most cases, I’m not the typical user. There are times when—from a best practices perspective—some element of a user interface may not appear to make sense. However, in doing a usability study with the target audience, I may find that these apparent issues don’t actually hinder users’ ability to complete key tasks. Conversely, even though a design may meet best practices, sometimes the typical users’ background, knowledge, or experience may prevent their fully understanding how to interact with the user interface as designed.”
Tailoring Your Answer to the Situation
“I’ve heard this comment many times in my interactions with project leadership,” says Leo. “I’ve given them several different responses, depending on how much data actually exists. Here are some example of responses that I’ve used:
questioning whether there’s a data gap—‘Thank you for your vote of confidence! Yes, I’ve seen a lot of similar situations in the past. For example, if I understand your situation correctly, it seems like Project X, which I worked on, was very similar. Sadly, not all situations map one to one, so I won’t know how much of Project X could actually apply to your project unless we get some baseline information. Do you have much user research or data that could jump start that baseline effort?’
using deflection—‘If I understand your situation correctly, it seems like Project X, which I worked on, was very similar.’ Assuming that they claim to have much of the user research completed: ‘Sadly, because that work was proprietary, we can’t just reuse it verbatim. We’ll want to do some light-weight calibration and validation of your situation to see what would work for you. Once we figure that out, I can reuse the appropriate design principles and patterns from prior projects as they apply.’
blending new data with known data—‘Thank you. I have worked on similar projects in the past. But like business strategy, marketing, and other efforts whose goal is to expand your business’s reach, we’ll need to do some general work to get the lay of the land for a new UX engagement. Just as you wouldn’t embark on a new business venture without some research to guide you, we wouldn’t want to start investing in design and development without some guidance. I’m not talking about a massive research effort, depending on what your team already knows….’
denying the honor—Thank you. While I have worked on similar projects in the past, the right approach for your project depends on what you expect to get done. I totally agree that, if this were limited to a best-practices consultation, I could offer you my expert opinion based on common design practices in this space. But, based on what I’ve heard you say, I think your project has more to it than that. Otherwise, I’d be happy to recommend some great resources for your team to use and save you a lot of money.”
Focusing on the Return on Investment for Research
“It’s important to note that user research is not always necessary,” replies Jordan. “Sometimes organizations already have research, data, or insights that could provide answers to the questions you’d intended to answer by conducting user research. I always recommend usability testing, but an alpha or beta launch could meet the same need. Neither of these approaches is required on every project. On the other hand, if a project does legitimately need user research or usability testing, you should focus on communicating the return on investment (ROI) of conducting research. Usually when organizations say something like ‘You’re the expert, we trust you,’ it means they hadn’t planned to conduct research or testing and are afraid of the cost or schedule implications. They’d rather not do anything that could put their timeline or budget in jeopardy.
“To avoid this situation, plan for research and testing from the beginning. Even if you’re not sure what form it will take, include it in your plan and earmark budget and time to do whatever you think may be necessary. If your clients are actually concerned about ROI, they’ll raise the issue during the planning phase of the project. Their argument might be something like: ‘The cost and time that conducting research and testing would require outweigh any value we’d get from it because we’ve already hired an expert who should know all of this.’
“As UX professionals, our expertise lies in information architecture, interaction design, usability testing, and user research—among other areas. We’re not capable of reading the minds of a particular organization’s customer base. But we are capable of conducting a study that we’ve designed to learn actual users opinions about things. Then we can use what we learn from our research to inform our decisions regarding information architecture and interaction design. Our motivation for doing user research and usability testing is to produce theories, then test them on real users. Each theory that you test should add to an organization’s understanding of their users. The value of gaining this understanding makes conducting user research worthwhile. If user research would deliver no value, you shouldn’t conduct it.
“To look at this another way, the intent of any product-design, Web site–design, or software-design project is either to optimize something or launch something,” continues Jordan. “If the intent is to optimize a user experience, we need to understand what users currently like and dislike about it. If the intent is to launch something new, we need to understand the most important elements of positive and negative user experiences with which the user has interacted in the past. So, regardless of the situation, user research and usability testing are often necessary.
“In the end, if a client is adamantly opposed to conducting user research, it may be an indication of organizational short-sightedness. Some organizations want user experiences that are good enough for the user; but that cater to the agenda of the organization itself. They won’t tell you this outright, but they’ll have an aversion to talking or listening to users or customers.
“Clients who have drunk their organization’s Kool-Aid often feel that their customers should be thankful to their organization—rather than their organization’s being thankful to their customers. These are date-and-dump clients who make good friends for a short time, on a specific project, but who aren’t visionaries. They’re clients who you’ll avoid once you’ve finished a project. They’re clients who drain you and make you feel horrible at the end of the day—because in the end, UX professionals want to create user-centric designs. Creating user-centric designs is hard when you’re working for an organization that puts its own wants and needs ahead of the users or customers.”
Communicating Why You Should Do User Research and Usability Testing
Tobias tells us, “There are five reasons why we need to involve users:
Without users, you cannot—by definition—do user-centered design. Without users, a good UX designer will create a design that follows best practices for screen design—patterns, font sizes, color contrast, layout and alignment rules, and other such things. That alone is of value, but it’s far from being enough.
Without learning about users’ needs, what requirements and needs are you designing against? Did you come up with them? In all cases where you’re not part of the target audience, that’s a bad idea. Unless you’re a total hero in fund management, oil exploration, rocket science, electron microscopy, or whatever kind of product domain you’re working on, you’re not representative of the users. Should you get user requirements and user needs from the stakeholders? How much do they actually know about the real context of use? How often have you heard a project sponsor say, ‘I used to do that job,’ but it was 20 years ago and his experience bears no resemblance whatsoever to the reality of today?
Without seeing users at work, you cannot fully understand the context of use. Even if you assume that the stakeholders have correctly acquired, analyzed, and documented valid, testable, and unambiguous user requirements, as someone on the receiving end of all this information, you may not have any experience with the product’s context of use. You’ll need to learn a lot of details and nuances about that context, but learning them from stakeholders would mean you’d get only a filtered version of reality. They might not be able to answer questions like: ‘How far away from the screen does the user stand when doing XYZ?’ or ‘Do the existing users really follow the protocol that the handbook dictates?’ Often, users don’t use a product in the way that stakeholders envisioned and planned—and the stakeholders may be completely unaware of this. I remember a project on which, during field observations, we invalidated a couple of key assumptions that the product owners had been 100% sure about.
Without understanding users’ needs, you cannot justify your design decisions. How could you possibly survive a design-review meeting with stakeholders if they have all the ammunition—the context knowledge—and you have only an empty gun? If they point to your design and ask, ‘Why is this UI element there? It’s not listed in the requirements!’ perhaps you’d say that you feel it would be useful. That’s not a very strong argument. But if you’ve been doing user research, you would be able to say: ‘This feature allows users to do XYZ, which we identified as a critical need when we were visiting users last month.’ If you’ve done usability testing, you could say, ‘During our first usability study, 90% of test participants did not understand how to do XYZ. This feature helps them do XYZ. We verified this during our second usability study; none of the participants had any issues with XZY.’ The best thing you can have as a UX designer is empirical evidence, because it trumps opinions and beliefs.
User research is fun, educational, and inspiring! For many, this is the best part of being a UX professional. It’s where the music plays that pulls our heartstrings.”
Dr. Janet M. Six helps companies design easier-to-use products within their financial, time, and technical constraints. For her research in information visualization, Janet was awarded the University of Texas at Dallas Jonsson School of Engineering Computer Science Dissertation of the Year Award. She was also awarded the prestigious IEEE Dallas Section 2003 Outstanding Young Engineer Award. Her work has appeared in the Journal of Graph Algorithms and Applications and the Kluwer International Series in Engineering and Computer Science. The proceedings of conferences on Graph Drawing, Information Visualization, and Algorithm Engineering and Experiments have also included the results of her research. Read More