Observing User Research

Practical Usability

Moving toward a more usable world

A column by Jim Ross

August 6, 2012

It’s usually a great idea to invite others to observe user research. There’s nothing more impactful than seeing people perform their tasks and encounter problems firsthand. Observation promotes empathy for users and the problems they face. For those who already believe in the value of user research, observation can strengthen their commitment to it. And actually observing user research can convert nonbelievers. On the practical side, observation saves time. A team that has observed your user research sessions can quickly discuss findings, formulate conclusions, and decide on what changes to make based on their direct observation of the sessions.

However, if you don’t manage observers properly, they can have a negative impact on your research. For example, halfway through a recent usability testing session, the project manager and clients interrupted the session three separate times by coming into the lab to inform me that the people listening in to the session through a speakerphone could no longer hear the session. These interruptions completely disrupted the session and made the participant and myself very uncomfortable. Evidently, my client had decided that the needs of the observers were more important than the research itself. Afterward, I sarcastically tweeted that there are only four reasons to disrupt a user research session: a fire, someone died, aliens have landed, or a zombie apocalypse. Someone later suggested adding, “Lava is coming,” which I think makes sense.

In this column, I’ll discuss how to get the benefits from allowing people to observe user research, while avoiding the problems that observation can cause.

Champion Advertisement

Why Do People Want to Observe User Research?

It’s natural that clients and project team members want to observe user research sessions. Often, you have a packed house of observers, but sometimes no one bothers to show up. Why do certain user research projects attract so many observers?

People Are Curious

It’s interesting to hear what people think about a product or service and see what they do when attempting to use it. Many clients and project team members have never had the chance to hear what their users think directly. Many clients are also curious about user research techniques themselves and how they can use them to learn more about their users.

There’s A Lot at Stake

More people come to observe when there’s a lot at stake in a project. High-profile projects, projects where there are disagreements among stakeholders, and risky or unusual projects attract more observers.

People Don’t Trust You

Some people want to observe the research themselves because they don’t trust you. They want to make sure you report the findings accurately and that the participants represent real users. Such people may include developers, designers, product owners, and other project team members who are defensive about their work. Sometimes clients don’t trust you to be alone with their customers or employees and insist on having a representative of the company present.

Clients Want to Learn How to Do Research Themselves

Some clients hire consultants to observe their techniques, so their internal team can perform the methods themselves on future projects.

Research Makes an Interesting Diversion

Who wouldn’t want to get out of the office, away from their typical workday routine for a day or two? Add in a comfortable atmosphere and free food, and it’s no wonder that you draw a crowd to the observation room.

Why Don’t People Observe User Research?

So, what accounts for those times when no one shows up to observe? Sometimes it’s the result of successfully evangelizing the value of user research. As the practice of user research within an organization becomes more frequent, sessions become routine, losing their luster, and people have satisfied their curiosity. People are busy and may attend only the research sessions for more important projects. Those who were formerly distrustful learn that user research isn’t out to get them, and they don’t need to be there to defend their work.

Research Is More Important Than Observation

It’s very important to make sure your observers understand that the research itself is far more important than their ability to observe the sessions. You must let participants know that they are being observed, and for many people, that’s a source of anxiety. However, most people overcome this initial discomfort if the observation is subtle and doesn’t interfere with the session. Observation is a nice-to-have that should never interfere with the comfort of participants, the naturalness of their behavior, or the ability of a researcher to interact effectively with participants.

Where to Locate Observers

The ideal place for observers is out of participants’ sight and hearing. Observing sessions from a separate room or viewing them through a Web conference and speakerphone are discreet methods that don’t provide a constant reminder to participants that they are being watched.

When Observers Are in the Same Room

If you must have observers in the same room as participants, take the following steps to minimize their impact:

Limit the number of in-room observers to as few as possible.
Introduce the observers to the participants. This allows the participants to see them as regular people, not mysterious, unknown observers with unclear agendas.
Set strict rules about where observers should sit.
Instruct observers to refrain from talking, making verbal utterances, or sending signals with their body language.
Provide a specific time when the observers can ask questions—preferably at the end of a session.
Give observers guidance on how to ask neutral, non-leading questions.

Observing In-Lab User Research

Usability labs and focus-group facilities are designed specifically to accommodate a large group of observers—whether they are observing usability testing, individual interviews, or focus groups. With one-way mirrors, cameras, video monitors, lounges, kitchens, and free-flowing food and drink, they make observation easy and comfortable.

However, labs are sometimes uncomfortable for participants. Bringing participants into a lab takes them out of their natural element and puts them in an artificial environment. Fortunately, there are ways to minimize their discomfort and make the experience more natural.

Mirrors Versus Cameras

Whenever possible, avoid facilities that use traditional one-way mirrors to observe participants. Mirrors instantly create the unnatural feeling of being scrutinized in a lab. Unlike more subtle observation methods that participants can quickly forget about, mirrors are a large, blatant, constant reminder that unseen people are watching and listening.

Instead, use modern, mirror-less facilities that have discrete cameras and feed video of the sessions to large monitors in a nearby observation room. You must still inform participants that they are being observed, but it’s much easier for them to relax and forget about the observers without the obvious and continuous reminder of a mirror.

If you must use a room with a mirror, set up the room so participants face away from the mirror, so they won’t have a constant reminder of its presence and can avoid the distraction of seeing their own reflection. Acknowledge the mirror and use humor to defuse their potential discomfort. You may be able to use curtains or shades to cover the mirror when no one is observing a session, or when you have cameras and a video feed set up.

Handling Observers’ Questions

Observers often think of additional questions that they would like to ask during a user research session. The problems are how to get those questions to the researcher and when to ask them.

Some observers want to be able to instantly ask questions during a session and may try various methods of getting them to the researcher—for example, sending em ails or instant messages or even coming into the room with a Post-it note. All of these methods are far too disruptive and difficult to handle when in the middle of a session. Plus, the questions observers want to ask may be unclear, inappropriate, lead participants, or require additional discussion before you can ask them.

The best way to handle observers’ questions is to have them note their questions throughout a session. At the very end of the session, excuse yourself from the participant and go to the observation room to ask the observers whether they have any additional questions. This allows you to discuss what they would like to know and think about the best way to phrase the questions. It also helps the observers to feel more involved in the research.

Observing Remotely

Using a Web conference session—for example, a WebEx meeting and a conference call—allows people to observe from anywhere in the world and provides a nonintrusive means of observing either remote or lab-based user research. It’s also a great way to get more observers involved. People who are too busy to leave work for a whole day of observation can more easily squeeze in time to view a session or two from their desk.

Beware of Having Too Many Individual Observers

When using Web conferencing software, participants may become intimidated if they see a long list of meeting attendees. When all of the observers are in different locations, this is unavoidable. But sometimes a long list results from different people watching at their own desks in the same building. If this is the case, set up a single conference room with a computer that remains connected to the Web conference all day. Then, people can gather in the conference room rather than cluttering up the meeting’s participant list, and you’ll avoid having people constantly joining and dropping from the conference call.

Use Mute, But Avoid Hold

Remind observers to put their phones on mute and, especially, not to put calls on hold. Muting their phones lets them avoid making any sounds that remind participants that there are people listening in—for example typing sounds, faint side conversations, or heavy breathing. The researcher, participants, and other observers certainly wouldn’t find hearing hold messages or music entertaining.

Observing User Research in the Field

While it’s easy to include observers when doing in-lab or remote research, it’s very difficult to accommodate additional observers when doing field studies.

Two Is the Magic Number for Doing Field Studies

Two people, including the researcher, are the ideal number to go on field studies. With more than two, you run into problems for the following reasons:

space—Most workspaces and homes can’t easily accommodate more than two people in addition to a participant.
participant comfort—Participants tend to feel comfortable with two people, but begin to feel outnumbered and intimidated if you bring along more.
efficiency—It’s helpful to have one person leading the session and another person to take notes, make recordings, and ask additional questions.
safety—For researchers, there’s safety in numbers—especially when you go into people’s homes.

Who Should Observe in the Field?

Since you’re limited to two people, who should those people be?

Obviously, one observer is the primary researcher. If the primary researcher is not also the designer, the second person should be the designer.

It’s far more powerful and effective for designers to observe user research firsthand rather than to wait for a report from the researcher. While clients and other project team members can wait for a researcher to complete the analysis of the findings and create a formal presentation, the designer is the person who actually has to use the research findings. There’s no substitute for seeing users, their tasks, and their context. This lets a designer immediately begin thinking about the problems and possible solutions.

Clients Should Not Observe Field Studies

So, if two is the ideal number of observers for field studies, and those two are the researcher and the designer, what happens when clients want to get in on the action? It’s understandable that they want to see the research firsthand, but including them is nearly always a bad idea—for several reasons.

There Would Be Too Many People

To include a client, you must either exclude any designer who isn’t the primary researcher or add more people to the sessions, which would become physically awkward and make participants uncomfortable.

Clients’ Presence Might Make Participants Uncomfortable

Because clients represent the product, application, or organization that you’re discussing during a research session, participants might not feel comfortable or able to speak freely or critically when clients are present. When clients don’t attend sessions, you often have the advantage of being an objective third-party in whom participants feel comfortable confiding. Bringing a client along alters that dynamic and can stifle open conversation.

Participants and Clients May Talk Above Your Level of Understanding

As an outsider, you can act dumb and ask participants to assume that you know nothing. You can get participants to talk to you at a beginner’s level. However, clients and participants often have the same level of knowledge about the product or organization you’re discussing, so participants may talk at that insider level rather than at the basic level you need to understand what they’re saying.

Participants Ask the Client Insider Questions

Because participants know that clients represent the product, application, or organization that is the subject of your research, they may ask them insider questions about problems, features, technical issues, or organizational issues that derail a session and send it down sidetracks that don’t provide any useful information.

How to Convince Clients Not to Observe Field Studies

Ideally, by making the points I’ve just discussed, you can present a convincing argument for why clients should not come along on field studies. Although this works for some clients, others may be more adamant. Sometimes involving them in planning the research and discussing the findings and giving them frequent status updates satisfies their need to be involved. You can also propose giving them video or audio recordings as a substitute for actually attending the sessions. And, if you do any remote sessions through a Web conference and phone call, you can allow the clients to attend those sessions, without having any negative impact on the sessions.

What If You Can’t Convince Clients Not to Observe Field Studies?

Sometimes despite all your efforts to persuade them, clients insist upon attending field studies. At some point, you may have to give in. However, you can still enforce some limits on their participation to minimize any problems it might cause, as follows:

Try to get them to attend only a few sessions to get a flavor of the research.
Allow only one client to attend each session. If there are multiple clients who want to attend sessions, insist that they divide the sessions among themselves.
Clearly define the clients’ role as silent observers. Give them a specific time at the end of each session to ask additional questions.
Inform participants in advance about who will be attending their session to give them time to get used to the idea of the client being there. Those who would feel too uncomfortable being observed by the client can excuse themselves from participating.
Look at the bright side. Though there are disadvantages to having clients observe field studies, clients are often good interpreters of users’ jargon, applications, organizational knowledge, and cultural issues. Between sessions, they can answer your questions and fill in some background on what participants are talking about.

Conclusion

User research findings are sometimes difficult to describe adequately in words, and there’s often no good substitute for seeing research sessions firsthand. However, it’s important for everyone to remember that the research itself is far more important than the ability of people to observe the research. So, you should take great care not to let the observation of research sessions affect the integrity of the research itself.

In User Research