Top

Should Your Entire Product Team Observe Usability Testing?

Ask UXmatters

Get expert answers

A column by Janet M. Six
March 23, 2015

In this edition of Ask UXmatters, our panel of UX experts provides a wide variety of responses that UX professionals might make to a client who declares that the rest of the product team need not observe usability testing.

You have the go-ahead from your client to conduct usability testing, but he tells you that other members of the product team don’t need to observe the test sessions. What should you do? Fire the client? Make the business case? Explain the consequences of the team’s not observing test sessions? Or maybe your client has a point. Do only certain team members need to observe usability testing? Would you make sure that the right people on the team view the right sessions? How often should people observe sessions? What are some other alternatives?

In my monthly column Ask UXmatters, our panel of UX experts provides answers to our readers’ questions about a broad range of user experience matters. To get answers to your own questions about UX strategy, design, user research, or any other topic of interest to UX professionals in an upcoming edition of Ask UXmatters, please send your questions to: [email protected].

The following experts have contributed answers to this edition of Ask UXmatters:

  • Carol Barnum—Director of User Research and Founding Partner at UX Firm; author of Usability Testing Essentials: Ready, Set … Test!
  • Dana Chisnell—Principal Consultant at UsabilityWorks; co-author of Handbook of Usability Testing
  • Caroline Jarrett—Owner and Director at Effortmark Limited; author of Forms that Work: Designing Web Forms for Usability; UXmatters columnist
  • Jordan Julien—Independent Experience Strategy Consultant
  • Cory Lebson—Principal UX Consultant at Lebsontech; President, User Experience Professionals’ Association (UXPA)
  • Gavin Lew—Executive Vice President of User Experience at GfK
  • Baruch Sachs—Senior Director, User Experience at Pegasystems; UXmatters columnist

Q: If your client declares, “The rest of the team doesn’t need to observe the usability testing,” what should you do?—from a UXmatters reader

“After many years of hearing this sort of thing,” answers Caroline, “and understanding how fundamentally wrong it is, I coined this phrase on Twitter:

“User researcher’s fallacy: ‘My job is to learn about users.’ Truth: ‘My job is to help my team learn about users.’”

“My first instinct is to think: Sack the client—especially since I’ve recently observed a project where the client said exactly that, refusing to allow key Business Architects, User Experience Designers, and others to observe testing. The result was: a full year of completely wasted work that could easily have been avoided if the rest of the team had observed the testing and designed with more understanding of real user needs.

“But counteracting that impulse: I’m privileged to be part of the GOV.UK Verify team at the moment. Our observation room for usability testing is always packed—but by no means by the whole team. Every team member must understand users, for sure, but that doesn’t always mean observing every single user research session for people like, say, back-end developers. Our team is part of the UK Government Digital Service (GDS), and we recommend two hours of observation every six weeks.” Caroline suggests that you read “Have You Had Your Recommended Dose of Research?

“So it could be that your client is exactly right,” concludes Caroline. “Some team members may have had their recommended dose of research quite recently, so don’t yet need to be topped up.”

“You have a few options,” advises Dana. “One is to leave. You’re probably at the wrong place. You might also point to the contract, where you specifically made attendance by the team a success criterion for the project. Didn’t do that? Then, it’s time to explain that if the team doesn’t observe usability testing, you can’t know whether the recommendations you’ll make are feasible and appropriate. You can hold your recommendations hostage, with your condition for their release being that all people on the team observe at least two usability test sessions.”

Make the Business Case

“The right answer depends on who your client is, as well as the nature of the project,” responds Jordan. “It isn’t necessary to observe all usability testing first hand—although it may be desirable to do so. There are many cases where an expert analysis is all the rest of the team really needs. When you hear your clients saying things like this, you usually need to make a business case for spending the extra man-hours observing test sessions. There are many ways of doing this, but the most persuasive arguments require an understanding of how your particular team is structured and how the organization will use the findings from the usability tests.”

Make Sure the Right People Observe

“I actually take this type of statement to heart because I do believe that not everyone does need to observe usability testing,” replies Baruch. “We first have to examine who exactly comprises the rest of the team. I would rather have the right people observe sessions than all of the people. You can pick your champions and let them spread the word about observing sessions, so it will be easier to get others to observe the next time. Chances are that team members who did not observe usability testing will get excited about doing it and tell your client that they want to observe the next round of testing. I have had exactly this situation happen—where the Project Lead wanted to see the usability testing, but did not want anyone else to observe. But once he experienced it, he saw the value of observing and wanted to involve the whole team. He even took credit for making it all happen. That was okay, because, in that case, usability testing grew organically from within the project team. An external consultant did not force usability testing upon them.”

“I tell them that’s fine,” answers Cory. “While, in an ideal world, all stakeholders would observe all test sessions, I know that this isn’t always feasible. In my consulting experience, it’s actually pretty rare for everyone to attend all test sessions. More typically, some stakeholders watch some sessions, but not every session has observers. As a standard part of my usability testing process, I make video recordings of the sessions and send the team links to download and watch the sessions whenever they want. Often, I’ll also include a note about which sessions I think they’ll find most interesting. Depending on the budget for a project, I may also create video clips that highlight key findings from the study.”

“We digitally record everything so people can watch sessions on their own time,” says Gavin. “But we know people often don’t watch them. When we get a negative response to a request for other team members to observe usability testing, we must communicate that it is important—not just to identify usability problems, but also to identify the source of the problems.

“For example, a problem might be that a user moved on before completing a task. The source of the problem may be the call to action for the concluding step or a lack of feedback that the user has not yet completed the task. Merely stating that a user didn’t complete a task is insufficient to get to the root of the problem. If your other team members are not present during test sessions, you have only your own descriptions of the problems and insights about solutions to convince the team and provide a clear rationale for necessary changes. So you’ll need to be more descriptive if many of your team members aren’t going to be there to observe the pain that led to a finding. And you’ll need to be crystal clear on the source of the problem if you’re to be able to fix it.”

Some Alternative Responses

“In a perfect world, everyone on a product team would want to observe usability test sessions and team leadership would support their doing so,” replies Carol. “But the reality, in many situations, is that tight development schedules and other commitments can make that difficult, if not impossible. So, I’ll suggest many different possible responses that you could use in such situations—for example:

  1. Could team members take turns observing sessions? At least one team member should observe each session. Each team member could observe just one session, but it would be better for each team member to observe at least two sessions to get a broader sense of users’ experiences.
  2. Could we set up remote viewing via GoToMeeting, WebEx, or similar collaboration tools, so team members could observe test sessions while multitasking and without having to leave their workspace?
  3. If the response to either of the first two options is no, ask whether the decision maker would be the team member observing. If the decision maker can guide the rest of the team to understanding what you’ve learned about the user experience and what the team needs to do to improve it, one observer is enough.
  4. If the answer to all of the above options is no, your next question should be whether you or, better yet, the team member who did observe, could present the findings to the full team. If time permits, you could support this presentation with video highlights to let the users speak about the issues for themselves. Alternatively, if the findings meeting has to happen right after testing concludes, you could do a joint presentation with the team member who observed the sessions to inform the rest of the team of your findings.
  5. If the answer to all of the above options is no, suggest that everyone on the team go through the session recordings to understand what you’ve observed and learned from users. However, I have to say that, at least in my experience, teams rarely follow up on this option—even when a client or team says they want to review the sessions. There just never seems to be enough time for this effort—in part because it occurs after the fact and the development timeline is already moving ahead.” 

Product Manager at Tom Sawyer Software

Dallas/Fort Worth, Texas, USA

Janet M. SixDr. Janet M. Six helps companies design easier-to-use products within their financial, time, and technical constraints. For her research in information visualization, Janet was awarded the University of Texas at Dallas Jonsson School of Engineering Computer Science Dissertation of the Year Award. She was also awarded the prestigious IEEE Dallas Section 2003 Outstanding Young Engineer Award. Her work has appeared in the Journal of Graph Algorithms and Applications and the Kluwer International Series in Engineering and Computer Science. The proceedings of conferences on Graph Drawing, Information Visualization, and Algorithm Engineering and Experiments have also included the results of her research. Janet is the Managing Editor of UXmatters.  Read More

Other Columns by Janet M. Six

Other Articles on Usability Testing

New on UXmatters