Sustainable User Research

By Jim Ross

Published: March 21, 2010

The title of my new column, Practical Usability, reflects my intention to explore practical and useful aspects of usability that are relevant to UX designers and user researchers, as opposed to focusing on an academic or narrowly defined vision of usability. In this column, I’ll use a broad definition of usability, encompassing all user research activities that contribute to a better user experience.

“In-person research is much better than remote research. But … now, more than ever, it’s important to determine when it’s feasible to save money and the environment by conducting more user research remotely.”

Traditionally, user research involves directly observing and talking with people in the context of their work or play. Either researchers travel to observe participants in their natural environments or participants travel to a usability lab or focus-group facility. How better to understand how people use a product or technology than to observe them using it firsthand?

Although lack of time and money for travel have always been barriers to conducting in-person user research, the current recession and concerns about global warming and wasted resources have pressured businesses to cut back on business travel and conduct more business remotely. Should user research be any different?

There’s no denying that, in most cases, in-person research is much better than remote research. But who are we to say that creating good user experiences is more important than the environment? Now, more than ever, it’s important to determine when it’s feasible to save money and the environment by conducting more user research remotely.

“Remote user research can be either moderated or unmoderated.”

Remote user research can be either moderated or unmoderated. In both cases, the participants and researcher are in separate locations. However, in moderated, remote user research, the researcher and the participants go through the research activity together virtually, while in unmoderated, remote user research, the researcher is not involved during the study.

In this column, I’ll discuss the advantages and disadvantages of moderated and unmoderated, remote user research. Then, I’ll reflect on some deciding factors for conducting either in-person or remote user research—or both in combination. Understanding all of these considerations can help you to decide when it’s most appropriate to use in-person or remote methods of user research—and if the latter, whether to do moderated or unmoderated research—or to combine both approaches and get the best of both worlds.

Moderated, Remote User Research

“During a moderated, remote user research session, the researcher and a participant typically communicate by phone and use software that lets the researcher observe the participant’s computer screen.”

During a moderated, remote user research session, the researcher and a participant typically communicate by phone and use software that lets the researcher observe the participant’s computer screen. In fact, Web-conferencing applications such as WebEx or software like TechSmith’s UserVue allow the researcher and participant to both see and control each other’s computers. It’s possible to record a user research session, using screen-recording software such as TechSmith’s Morae and a microphone. Using a Webcam to view the participant is also an option, as I’ll explore later.

Advantages of Moderated, Remote User Research

First, let’s consider the advantages of moderated, remote user research.

Eliminating Travel and Saving Time, Money, and the Environment

Remote user research eliminates travel, thereby saving time and money. You can avoid wasted time in airports, there’s no need to rent or reserve a usability lab or other research facility, there’s no wear and tear on your equipment, and usually, incentives can be lower, because participants don’t have to travel to your lab’s location. Obviously, reducing travel offers important environmental benefits, too.

Including Participants from Anywhere in the World Is Easy

“Remote research lets you include people who would otherwise be left out, so you can have a more representative sample.”

If you conduct only in-person user research, budget and time considerations often limit you to using only local participants or traveling to just a few locations. Remote research lets you include people who would otherwise be left out, so you can have a more representative sample. Moreover, it’s often easier to recruit participants when they don’t have to travel to a usability lab to participate.

Conducting Usability Testing Sessions That Are More Representative of Real Situations

In remote usability testing, participants use their own computers in their home or workplace, where they typically feel more comfortable than in a usability lab. In such environments, you can witness participants’ Internet network speed, browser choice, bookmarked sites, and so forth, as well as the natural interruptions and interactions that occur in real-world environments.

Thinking Aloud Is More Natural for Participants

“Participants are more likely to communicate with you verbally and explicitly narrate their actions as they move through a task, which gives you a better understanding of what they are thinking and doing.”

Because you can’t see the participants, it’s even more important than usual for them to think aloud as they work, so you can understand what they are doing. When a participant stops thinking aloud during an in-person testing session, you can usually see why—he or she may be reading, confused, in deep concentration, or frustrated and about to give up. But in a remote session, you miss these visual cues.

Fortunately, because participants can’t see you either, the need to think aloud is obvious, and doing so comes more naturally to participants who are working remotely. Participants are more likely to communicate with you verbally and explicitly narrate their actions as they move through a task, which gives you a better understanding of what they are thinking and doing.

Focusing the Researcher on Observing Rather Than Interacting

“Greater focus can lead to richer insights and more fruitful discussions with participants.”

Engaging with participants in remote locations removes the dynamics of body language and eye contact, allowing you to focus on your computer screen and your notes. Greater focus can lead to richer insights and more fruitful discussions with participants. Plus, during a remote session, you can log events on your computer, saving time, making data analysis easier, and—as an extra environmental benefit—not using up paper.

Disadvantages of Moderated, Remote User Research

Now, let’s look at the disadvantages of moderated, remote user research.

Seeing Participants and Their Context Isn’t Possible

“During a remote user research session, the only context you can see is a participant’s computer screen. You miss seeing the participant’s facial expressions and body language….”

Since participants’ computer interactions are the focal point of user research, it’s easy to imagine how you could successfully conduct remote usability testing. But remote contextual inquiry is an oxymoron. How can a user interview be contextual when you can’t see the user’s context firsthand?

During a remote user research session, the only context you can see is a participant’s computer screen. You miss seeing the participant’s facial expressions and body language, as well as any interactions with other people, the surroundings, or supplemental materials—for example, Post-it notes, printouts, calendars, or documents.

You can use Webcams to capture participants’ facial expressions and body language. But sending a Webcam to each participant is expensive and coordinating the installation of Webcams and software for multiple participants can be difficult, and seeing participants’ facial expressions through the typically delayed video a Webcam provides is rarely worth the effort. Sending disposable cameras to participants so they can photograph their surroundings and materials is a much less expensive way of filling in some of the details you miss when doing remote user research.

Establishing Rapport with Participants Can Be More Difficult

“You’ll have to work harder to establish a sense of rapport with participants. ”

Distance and the technology for conducting remote research can add a level of impersonality to the interactions between you and participants during test sessions. You’ll have to work harder to establish a sense of rapport with participants.

Maintaining Control Over the Situation Is More Difficult

Because you’re not physically present, you have much less control over a research session. Frequently, coworkers, family members, phone calls, instant messages, and even pets interrupt participants. Although these interruptions provide a sense of participants’ real lives, which is part of what you want in a contextual inquiry, they can completely disrupt tasks during usability testing and ruin your studies’ results.

Participants May Invite Others to Attend

For some reason, participants in remote user research occasionally invite other people to attend a test session—often a coworker they believe might have useful information. Perhaps the use of a conference call and Web-conferencing software causes participants to think of a user research session as a meeting. If you’re lucky, the participant will inform you that the extra person is present. However, your inability to see who else might be lurking in the background makes it difficult to eliminate the presence of these extra attendees.

Web-Conferencing Software Can Have Technical Problems

It happens less often than it used to, but some participants still have problems using Web-conferencing software. Those with a slow Internet connection may not be able to connect and share their screen. Slow connections can also delay your view of a participant’s actions.

Unmoderated, Remote User Research

“While moderated, remote user research separates the researcher and participants by distance, unmoderated, remote user research separates the researcher and participants by both time and distance.”

While moderated, remote user research separates the researcher and participants by distance, unmoderated, remote user research separates the researcher and participants by both time and distance. Although unmoderated, remote user research offers the advantage of letting you reach more participants, expending less time and money, it also brings the major disadvantage of further separating the researcher from participants.

In unmoderated, remote user research, the researcher sets up a study, then participants complete the study on their own, without any involvement of the researcher. Finally, the researcher views the results collected by the software the participant used during the study.

Surveys are the oldest and most familiar type of unmoderated user research, while tools for online card sorting and self-guided usability testing are more recent inventions. Today, a wide variety of online tools combine questionnaires, usability test tasks, and Web analytics. These tools can track the paths participants took, how long they spent on each page, and where they clicked during a test task. Unmoderated, remote user research is a useful addition to the researcher’s toolkit, but it, too, has advantages and disadvantages.

Advantages of Unmoderated, Remote User Research

Next, let’s consider the advantages of unmoderated, remote user research.

Tools for Unmoderated, Remote User Research Do a Lot of the Work for You

“Unmoderated user research tools automatically collect, analyze, and present data in useful visualizations, eliminating time-consuming data collection, organization, and analysis tasks for you.”

After setting up a study and contacting the participants via email, you are free to work on other things, while the participants complete the study on their own time. Unmoderated user research tools automatically collect, analyze, and present data in useful visualizations, eliminating time-consuming data collection, organization, and analysis tasks for you.

Including a Large Number of Participants Is Easy

Because participants complete a study on their own time and unmoderated user research tools automatically collect and present data for you, it’s easy to include hundreds or even thousands of participants at a fraction of the cost and effort of in-person research. This is especially helpful for activities that require a large sample size—such as card sorting, for which you need 30 or more participants to generalize the results to a larger user population. [1]

Including Participants from Anywhere in the World Is Easy

“With no geographical limits and higher numbers of participants, you can aim to get a truly representative sample of your users.”

As with moderated, remote user research, you can easily reach participants anywhere in the world. Again, this enables you to include many participants who would otherwise be left out because of time constraints and travel costs. With no geographical limits and higher numbers of participants, you can aim to get a truly representative sample of your users.

Eliminating Travel and Saving Time, Money, and the Environment

Unmoderated, remote user research also eliminates the need for travel and provides the same savings of time and money as moderated testing does. Again, this eliminates the negative environmental impacts of travel.

Disadvantages of Unmoderated, Remote User Research

Finally, let’s consider the disadvantages of unmoderated, remote user research.

Unmoderated, Remote User Research Provides Little Qualitative Information

“Unmoderated, remote user research is best suited to gathering quantitative information.”

While moderated, remote user research prevents you from observing a participant’s context, unmoderated, remote research additionally makes any dialogue with the participants impossible. Although most unmoderated, remote user research tools support survey questions or allow participants to make comments, such responses tend to be very shallow and limited in comparison to in-person discussions. Because of this, unmoderated, remote user research is best suited to gathering quantitative information.

Unmoderated, Remote User Research Tools Have Limited Scope

The best unmoderated, remote user research tools tend to do only one thing well. For example, online card sorting tools communicate how users would organize and label a set of information, but they don’t do much beyond that. Similarly, most online usability testing tools are good at things like testing findability tasks and posing survey questions, but they can’t capture the complex interactions and user behavior you could observe either in-person or through remote, moderated usability testing.

Unmoderated, Remote User Research Offers the Least Control Over Participants

“With no researcher present during a test session, participants are more likely to get distracted by interruptions, take breaks, or put less serious thought and effort into completing a study.”

With no researcher present during a test session, participants are more likely to get distracted by interruptions, take breaks, or put less serious thought and effort into completing tasks. With many online tools, you have no way of knowing whether a participant suddenly got distracted by a phone call, went out for a snack, or just took a long time to complete a task. Obviously, this can seriously throw off the results of your study. Without a moderator, test sessions need to be much shorter to ensure participants can complete them before losing interest.

There’s No Moderator to Prevent Errors and Misunderstandings

Because there’s no researcher to explain the study or help participants complete tasks when they get stuck, unmoderated, remote user research tools must provide extremely clear instructions and have very usable user interfaces to prevent bad data from corrupting a study’s results.

Deciding Between In-Person or Remote User Research

“Choosing between in-person or remote user research depends on what you’re trying to accomplish. For many studies, it’s very important to directly observe and have in-depth discussions with participants.”

Choosing between in-person or remote user research depends on what you’re trying to accomplish. For many studies, it’s very important to directly observe and have in-depth discussions with participants. For example, technology workflow studies require seeing the participants’ environment, materials, and people with whom they interact. However, for other activities such as card sorting, using automated tools offers time-saving benefits, because they can collect, analyze, and present the data from a large number of participants, in many different locations.

To determine whether remote user research might be acceptable, first define your research goals, then ask yourself the following questions:

  • How many participants do you need?
  • Where are the participants?
  • Is it necessary to include participants in many locations?
  • How much time do you have to conduct the user research?
  • How much money do you have for user research?
  • Are you trying to gather qualitative or quantitative data?
  • How important is it to observe the context of use around what you’re studying?
  • How important is in-depth discussion with participants?
  • What will you miss by not being with participants in person?

Consider Doing Both Remote and In-Person User Research

“Remote and in-person user research can complement each other, providing the best of both worlds.”

Of course, you don’t have to choose one method of user research over the other. Remote and in-person user research can complement each other, providing the best of both worlds. For example, you can conduct remote usability testing or contextual inquiries to include participants in distant locations and in-person sessions with local participants.

To make up for the lack of qualitative data from unmoderated card sorting or usability testing studies, conduct some in-person card sorts or test sessions to gather some qualitative feedback or further investigate issues you’ve discovered during the unmoderated testing.

Both in-person and remote user research have their appropriate place in the usability professional’s toolkit. By using these techniques wisely, you can take advantage of their strengths, minimize their weaknesses, and save time, money, and the environment by eliminating unnecessary travel.

Reference

[1] Tullis, Tom, and Larry Wood. “How Many Users Are Enough for a Card-Sorting Study? Proceedings UPA 2004, Minneapolis, MN, June 7–11, 2004. Retrieved March 3, 2010.

1 Comment

Yeah, I use the RHUB remote user research appliance for conducting remote sessions with my participants. And you are right. It allows host and participant to communicate by phone typically and allows the host to observe the participant’s computer screen.

Join the Discussion

Asterisks (*) indicate required information.