What Is the Client’s Role in User Research?
Published: March 4, 2013
You’ve just signed a contract with a consulting firm, engaging them to provide user research services. You’ll write checks to pay for those services. You’ll attend meetings. But what else does your role as a client on a user research project entail?
In my column, Practical Usability, I usually discuss user research from the researcher’s point of view, but clients play an important role as well. Successful user research involves close collaboration between clients and researchers to ensure that the research focuses on the right issues and provides acceptable recommendations. So, in this edition of my column, I’ll speak directly to the clients of researchers about the steps they should take to stay involved throughout a user research project and ensure its success.
Define the Problem
As a client, the first thing you should do is define the problems you want the research to solve. Before the project officially starts, gather the main stakeholders—the people who have a stake in the project’s outcome—to answer these questions:
- What are the problems that you want to solve?
- What do you already know about the problems?
- What do you want to learn through doing user research?
- When do you need to have the findings?
- What type of deliverable would be most effective for your organization—a detailed report, a formal presentation, or a quick and informal discussion of the findings?
- What are you going to do with the results?
- What obstacles might stand in the way of implementing the findings?
- What would you consider to be a success for this project?
To properly focus the research plan, you’ll need to convey this initial information to the researchers at the start of the project.
Kick Off the Project
At the kickoff meeting, present the information that you’ve gathered about the current situation and the problem you need to solve. Invite stakeholders who can help provide insights into the following:
- reasons for the project—that is, the problems you need to solve
- any prior research and the information you’ve gathered on the issues
- user groups, their characteristics, and the tasks they perform
- business or organizational goals for the research
- what you want to learn from the research
If there is an existing application, provide a demonstration of the main tasks that users perform with it.
Ease into the Subject Matter
Especially for researchers coming in from the outside, one of the most difficult parts of beginning a new research project is learning the subject matter. As the client, you and the other stakeholders are the subject-matter experts. Educate the researchers and designers by providing a high-level overview of your business, as well as any background documentation that would help the project team to get up to speed quickly.
Provide Enough Time for Research
Unless there’s an important deadline that you must meet, be flexible about scheduling the research—and don’t create artificial deadlines. Unnecessary time pressure often results in poor-quality findings. If you’re going to spend a lot of time and money on user research, allow enough time to both prepare for and conduct the research properly and thoroughly analyze the results.
Select Stakeholders for Interviews
After the kickoff meeting, researchers often gather additional information and perspectives through stakeholder interviews. Select the stakeholders who should participate in these interviews. Remember, stakeholders are not users. As the term implies, they’re people who have a stake in the project, and their role is to represent business needs.
At the beginning of a research project, no one knows more about your product’s users than you do—especially when the researchers aren’t part of your organization. So your helping to recruit the right participants is crucial to ensuring the success of the research. Describe the characteristics that define each of the user groups to the researchers. They will use that information to create a screener document comprising a list of questions that potential participants will answer to determine whether they’re the types of people you’re looking for.
On some projects, the researchers or a recruiting company will recruit the participants. However, when the participants are your employees, customers, or members, with whom you have existing relationships, it usually makes sense for you to do the recruiting. People are more likely to participate in research when they get a request from someone in their own company, a company with whom they do business, or an organization of which they’re a member. Ideally, you already have access to lists of employees, customers, and members.
Delegate Recruiting and Scheduling
However, even with the advantages of an existing relationship, it’s not always easy for a client to recruit participants. Consider whether you have the time and the necessary skills to contact, screen, and schedule participants. If not, delegate this task to someone who does. If you don’t have time, but want someone in your organization to handle recruiting and scheduling, an administrative assistant might be able to handle this task.
Get a Description of the Study from the Researchers
Ask the researchers to provide a description of the study and what participating in the research requires. You can use this description in your email messages or phone calls to prospective participants. When sending email messages to participants, include your name and contact information to make them seem more legitimate. Very few participants will actually contact you, but it makes them feel better if they know there’s someone they can contact.
Ensure That You Get the Right Participants
Be careful to recruit the right types of participants. I’ve had clients make the mistake of recruiting experts—the people who know the most about the system—rather than typical users. For example, they might recruit system administrators or department managers because they know so much about the system and can provide the most information. But these experts are far from being typical users, so involving them skews the results of the research. Be sure your researchers provide you with information about how to recruit participants, as well as details about the types of people to recruit, then follow their instructions.
Schedule the research sessions logically. Allow time for the researchers to get from one place to another between sessions. Include time for meals and breaks—plus some extra time, in case sessions go longer than expected. Scheduliing sessions within a tight time frame generally works better than spreading too few sessions over too many days. Schedule sessions logically by their location. Needlessly sending a research team on multiple trips would waste a lot of time, and you’d pay a lot in travel expenses. For example, I once drove from Philadelphia to Reston, Virginia—a seven-hour, round-trip drive—for one 90-minute contextual inquiry.
Review the Discussion Guide
The researchers will prepare a discussion guide for the sessions. This is a document that lists the tasks that the participants will perform and the questions that a researcher will ask the participants. Carefully review the discussion guide in advance of the sessions to make sure the research will obtain the information you need. Discuss any changes that you’d like to make with the researchers.
Nevertheless, it’s important to realize that a discussion guide really is just a guide. Research sessions are unpredictable and never go exactly as planned. Researchers need to be flexible and make changes on the fly to capitalize on unexpected opportunities that arise. You have to trust the researchers to use their judgment regarding when to stick to the guide and when to make changes to their approach or ask additional questions.
Observe the Research
There’s simply no better way for you to understand your users and the problems they face than listening to them and observing them perform tasks. So, if you can observe the research sessions without affecting the comfort of the participants or the quality of the research, you should. Sure, you’ll get a final report or presentation, but that doesn’t offer nearly the same value as observing the research firsthand. Even watching video clips isn’t the same as being there in person. If you really can’t attend in person, at least try to observe and listen to the sessions using Web conferencing tools.
Encourage Others to Attend
Encourage others in your company to attend the research sessions as well. Include people who might not know much about usability and user research. There’s no better way for them to gain an understanding of and empathy for your users and the problems they face.
This may sound obvious, but it’s more difficult than you might think to focus your attention on observing the sessions and make sure that everyone who is observing pays attention throughout the sessions. After watching several repetitive sessions, it’s all too easy to lose focus and get distracted by checking email, looking at Twitter, reading blogs, taking calls, or trying to get some work done. To keep everyone focused, take notes and discuss what you’re seeing with the others in the room. List major issues on a whiteboard.
Keep an Open Mind
You’ll probably come into the research with some existing hypotheses and preconceived notions, but don’t let those close your mind to information that differs from them. People tend to dismiss information that doesn’t fit with their beliefs. At worst, observers might blame a participant for not fitting the user profile that they expected, allowing them to dismiss everything the participant says and does. Try to remain objective about what you’re seeing.
Don’t Jump to Conclusions
During the research sessions, the researchers see patterns developing, but they wait to gather all of the data before analyzing it and making any definite conclusions. In contrast, observers sometimes jump to conclusions after watching only a few participants, then view the remaining participants’ sessions as either reinforcing their conclusions or being exceptions that they can dismiss. Although it’s a good idea to list and discuss the themes and issues that you see emerging throughout the sessions, keep an open mind, and don’t come to your final conclusions until the research is over.
Resist the Urge to Make Changes During Testing
The best way to drive researchers crazy is to make a lot of changes to the discussion guide and the prototype during usability testing. The more changes you make during testing, the more opportunities you create for error and inconsistency.
Changing the discussion guide is likely to confuse and throw off the researchers, who already have a great many things that they must do and remember during the sessions. If, on top of all this, you make changes to the guide, you’d also be making them either type up the revisions to the guide, then reprint it; manually write the changes on every printed copy of the existing guide; or try to remember all of the changes.
Making changes to the prototype or the discussion guide also risks affecting the integrity of your test results. If the researchers give task A to participants one through three, then give Task B to the remaining participants, you’ll have seen only three participants perform Task A. You’ll never know what the remaining participants would have done with that task. Similarly, if you change Page X to Page Y after seeing three participants’ reactions, you’ll have made a design decision based on the results from only three participants. Remember, a good researcher gathers information objectively and starts noticing trends and issues, but waits to make any final conclusions until he’s gathered all of the data. You should apply the same philosophy to making changes during testing.
The exception to this general rule is when there are problems—such as a poorly worded question in the discussion guide that is confusing participants or an unexpected problem with the prototype that affects how it works. In such cases, it makes sense to make the changes that are necessary to correct the problems. But resist the urge to make other changes while you’re at it.
Arrange a Good Time to Talk with the Researchers
Some observers expect to be able to contact the researchers during the research sessions—perhaps to give them instructions or suggest additional questions to ask participants. But doing so would be extremely distracting to the researchers and disruptive to the sessions.
During each session, researchers must observe, listen, understand, think about the next question to ask, maintain a neutral demeanor and body language, make good eye contact, and establish a good rapport with the participant. Asking the researchers to take questions from the observers as well would require them to handle the additional tasks of noticing questions that come in, understanding what those questions mean, thinking about how and when to ask the questions, and rephrasing the questions in an unbiased manner.
A better way to handle any additional questions that you might want to ask a participant is to save them till the end of each session. At that appointed time, the researchers can come to the observation room, note your additional questions, then return to the participant and ask those questions.
Never Interrupt a Research Session
Unless it’s a true emergency, never interrupt a research session. The integrity of the research is far more important than your ability to observe a session or communicate with the researchers.
Don’t Observe Field Studies
While there are great benefits to observing user research in the lab, it’s best if clients don’t observe field studies. Because field studies involve observing participants in their natural environment at home or at work, there is no observation room. So there is no way for clients to observe the sessions without their presence affecting the research.
Because participants perceive the clients as representing the product or system that is the subject of the research, they often feel uncomfortable when they’re present. As a consequence, they may censor what they say and do during a session. In addition, the limited physical space in a participant’s environment makes it difficult to include extra observers in a field study.
The quality of the research itself is far more important than your ability to observe. If you absolutely must observe field studies, schedule a few extra sessions and attend only those sessions to get a feel for the research. For the remaining sessions, trust your researchers and rely on their findings.
Feed the Researchers
During the research sessions, the most important people to keep mentally and physically sharp are the researchers. Doing user research is very mentally demanding and can be exhausting. Researchers who are fatigued or hungry won’t be in the top mental condition that is necessary to effectively facilitate the sessions and notice everything that occurs during them. So don’t schedule too many sessions in a day, provide enough time for breaks between sessions, let the researchers relax during the breaks, and make sure the researchers get fed.
Allow Enough Time for Analysis
A common lament of researchers is that there’s never enough time for analysis. Few people other than researchers understand what the analysis of research data involves. For clients, it often seems like there’s a long wait between the end of the research sessions and the presentation or report on the findings.
A thorough analysis is very valuable, but also very time consuming. It typically involves listening to or watching the recordings of all the sessions, while taking notes; combining related findings, organizing the findings, looking for patterns, coming up with recommendations, and creating the deliverable.
How much time is enough for analysis? This depends on several factors:
- number of participants
- length of the sessions
- type of research—a well-defined activity like usability testing takes less time to analyze than something more free form, such as field research
Don’t set arbitrary deadlines. Yes, there will be times when you do have tight deadlines, but otherwise, be sure to allow enough time for analysis. The more time you allow, the more detailed and higher quality the results of the analysis will be. If you’ve spent a great deal of time and money recruiting participants and conducting the research, rushing through the analysis to get the deliverable out fast is a tremendous waste of time and resources.
Wait for the Findings Before Making Changes
Some people make the mistake of implementing changes immediately following the research sessions, without waiting for the researchers to present their official findings and recommendations. Unless there is an urgent problem that needs to be fixed immediately, it’s best to wait. While you’re waiting for the researchers’ presentation, it is okay to discuss what you’ve observed and talk about some possible solutions. But don’t take any actions until you’ve reviewed the final deliverable.
Ensure That Everyone Attends the Presentation
Schedule the presentation of the research’s findings far enough in advance to get it on everyone’s calendar, and make it clear that attendance is mandatory. Those who miss the presentation won’t be able to catch up simply by reading it—and usually won’t read a report—so make sure that everyone attends. Otherwise, they’ll miss out on having a deep understanding the research and its recommendations.
Implement the Recommendations
An unfortunate and surprisingly common outcome of user research projects is that the research goes really well and everyone agrees with the findings and recommendations; but after the final presentation, the results just sit on a shelf and nobody ever actually implements them. To avoid this fate, you need to have a plan in place for implementing the recommendations.
After the presentation of the research findings, schedule an additional meeting to review the recommendations, assign priorities to them, estimate the level of effort that implementing them would require, determine which items to focus on implementing first, and assign people to be responsible for implementing those changes. Then, follow up periodically to see which recommendations the development team has implemented.
Conduct Iterative Research
At the end of a research project, don’t assume that you’re finished. You should do user research iteratively, evaluating and validating design changes and improving your understanding of your users. Perform field studies every so often to add to your knowledge of users. Create personas that embody the knowledge that you’ve gained. As you do additional research, add any new information that you discover about users to the personas.
Clients should take an active role in a user research project. User researchers are experts in understanding users, their tasks, and their needs. Clients are experts in their business needs and goals. A successful user research project needs to balance both business and user needs. Clients are instrumental in achieving that balance. When clients adopt the role of an active partner in research, a successful research project is the result.