User Research Is Unnatural (But That’s Okay), Part I

Practical Usability

Moving toward a more usable world

A column by Jim Ross
April 5, 2011

Thanks for coming in today. I appreciate your taking time away from work in the middle of the day to drive downtown, park, and find your way to our office. Most people wouldn’t do that, but the recruiting company said you’ll be getting a big check, and who can’t use some extra money these days?

Let’s go into this room here. Oh, the big mirror? Yes, as you can imagine, there is a room full of people behind it who are watching and listening to us, but try not to think about that. We also have some people who couldn’t be here who are listening in through this speakerphone, and we’ll be recording the entire session through these cameras and microphones. But don’t worry; no one outside the project will see the videos. Ha ha! No, they won’t end up on YouTube.

Let’s sit down at this computer. I’ll show you a Web site and give you some things to find and do. They may not be things you would normally do, but try to imagine that you really want to accomplish these tasks.

Champion Advertisement
Continue Reading…

By the way, this is just a prototype. It doesn’t have the images or text in it yet. No, that’s not Spanish, that’s just placeholder text. Since it’s a prototype, not all the links and functionality work, so at times, I’ll just describe what would normally happen, and you’ll just have to imagine what the site would really do.

Just try to do these tasks as you normally would if you were doing this at home. Except, try to think aloud, telling me, step by step, what you’re trying to do and what you’re thinking. That might be kind of awkward, but if you stop talking, I’ll just keep pestering you to think aloud, or I’ll start asking questions. At some point, it will probably become difficult to perform these tasks, think aloud, and answer my questions all at the same time, but try your best anyway.

Meanwhile, I—and all of those unseen observers that I mentioned—will be watching and listening to you and taking notes. I’ll ask you a lot of questions, but I won’t be able to answer many of your questions. You may not know whether you’ve correctly accomplished the tasks, but I’ll know, and I’ll make note of that. You’ll probably know when you’ve done or said something really interesting, because you’ll notice me furiously jotting down notes.

But remember, this isn’t a test of you. You’re helping us test out the Web site. So just act naturally and show me how you would really do these things, if this were a real Web site, these were tasks you would normally want to do, and you were at home in your normal environment. Any questions?

From the perspective of a participant, user research is not very natural. We ask participants to try to act naturally in the artificial environment of a lab, or we impose ourselves on their environment and hope our presence doesn’t affect their behavior. We often forget how unnatural user research can be and what effect it can have on participants.

Why Does It Matter That User Research Isn’t Natural?

It matters because we want to observe natural behavior, and we want to get participants’ honest feedback. If they feel uncomfortable or as if they are doing something artificial, it affects their behavior and responses.

Experiencing user research from the participant’s perspective can be eye opening. I had my first bad experience as a participant about 20 years ago, long before I became involved in usability. I was traveling to France for the first time and had a layover of a few hours at Heathrow Airport in London. I was young, jetlagged, confused, alone, and very self-conscious about being in Europe for the first time. After I had cleared customs, a woman asked me whether I would participate in a study of what people do while waiting in the terminal. It sounded interesting, so I agreed—not realizing they would give me a very large number to hang around my neck, so they could observe where I went. Of course, I was completely embarrassed and sat, as inconspicuously as possible, reading a book for three hours. The poor design of this observational study completely affected my behavior. I still wonder whether any of the other numbered participants were brave enough to act naturally that day.

What’s Unnatural About User Research?

User research studies aren’t completely natural, but that’s okay. As long as we understand and acknowledge the ways in which user research is unnatural, we can minimize the impact this has on participants’ behavior. In this first part of a two-part Practical Usability series, I’ll examine ways in which user research can be unnatural. Part II will recommend ways of minimizing or eliminating these unnatural aspects of user research, allowing you to observe more natural user behavior.


The most natural user research would take place without participants’ even knowing we were studying them. However, in most cases, this would be highly illegal and unethical unless you were working for an espionage or law enforcement agency. The principles of informed consent require that participants understand your study and consent to participate.

The People We Recruit

The first place unnaturalness creeps into user research is in how well the people we recruit for a study represent the typical user. Unfortunately, typical people don’t usually volunteer for user research. Those who do volunteer are sometimes in it only for the money, or they just enjoy being a participant. Such professional research participants may lie to get through a screener and may be either uninterested or overly eager to please, in the hope that they’ll get invited back for future studies.

The Incentives We Provide

In addition to attracting professional research participants who want to supplement their income by participating in research, incentives influence behavior. Research that has evaluated the effects of providing incentives on user research has shown that incentives cause participants to feel obligated to put more effort into their participation.

Rewarding Good Participants

Well, what’s wrong with getting people to put more effort into a study? People who are natural when thinking aloud and can eloquently express themselves are considered great participants. They provide great sound bites, feature heavily in highlight videos, and get invited back for future research. Observers are more likely to remember them and place more weight on their comments, while dismissing quieter participants as bad or unrepresentative participants.

By screening for outgoing, articulate people, we reward people who we think would be good participants and punish those who would make bad participants. Professional research participants learn this and adapt their behavior, so they’ll get invited back. Because of this, introverted and quiet people are often underrepresented in research—even though they may represent a significant part of the user population.

Where the Research Takes Place

User research can take place either in the lab or in the field.

Lab Studies

We often ask participants to leave the place where they would normally do something we’re studying and come to our artificial usability lab or focus group facility. Why? Usually, we do this for the convenience of the researchers and observers. That’s where we’ve got the recording and observation equipment set up; we can save time and money on travel; everyone can come to one place to observe the research; and we can get the study done more quickly. Meanwhile, the participants go through the inconvenience of taking time off from work or their personal lives to come to us and participate in the study.

Field Studies

Most people feel much more natural and comfortable when being observed at work or at home. They are on their own turf, using their own technology, with their usual tools and artifacts available. Because the purpose of field research is to observe natural behavior, this eliminates most of the artificiality of lab-based research.

However, even field research is not completely natural. It’s not an everyday occurrence to have several people come into your home or workplace who want to observe you, interview you, record what you do and say, and take photos. Your home is a much more personal space than your workplace, and you have a high expectation of privacy there.?With several strangers in their home, we can’t expect people to act the same they would when they’re alone.

Explaining the Research

Researchers work hard to create a casual rapport with participants. We strive to seem like relatable, regular people taking participants through a casual conversation, instead of cold scientists examining subjects in a lab.

The Explanation of the Research

Unfortunately, our warm, friendly tone changes when we explain the research. The rules of informed consent sound like a legal disclaimer or the reading of a suspect’s Miranda rights: “The session should last about an hour, but you have the right to stop the session at any time for any reason. With your consent, we will record the session, but you have the right to request that we stop recording at any time…” If this is handled improperly, an abrupt change from a friendly to a formal tone can result in a very awkward transition that undoes your careful work to put a participant at ease.

In graduate school, I was advised to read exactly the same instructions verbatim from a cue card to every participant to avoid bias. I agree that doing this with every participant would help avoid bias, but does that consideration outweigh the impact of our greeting participants with a robotic reading of what sounds like a legal disclaimer?

The Consent Form

A consent form is integral to informed consent, but the need to read and sign a formal document sets a formal tone for the session. When we ask people to sign what looks like a legal document or waiver, it puts them on edge. They might think, What am I giving away in this session? So, what we do with the intention of empowering and informing participants often seems as if we’re asking them to waive their rights and protect our organization.

The Things We Ask Participants to Do

In usability testing, we sometimes ask participants to do things they wouldn’t normally do, using software or Web sites they wouldn’t normally use. We ask them to imagine a scenario and show us what they would do in that situation. Ideally, if we have recruited the right participants, the things we ask them to do aren’t that far from reality, but sometimes they’re a stretch.

If you ask participants to do things they aren’t comfortable doing, you won’t get realistic behavior. If you do manage to find people who aren’t embarrassed by doing things that would embarrass most people, you have to wonder how representative those participants are. For example, a marketing person once described some in-home market research he did for a shaving products company in which five people crowded into bathrooms to observe how women shave their legs. I don’t know how they found women who were willing to volunteer for that kind of study, but I have to wonder how representative those participants were.


Although there are great benefits to early usability testing, low-fidelity prototypes are not what is familiar to the average person who uses software or Web sites. The lower the fidelity of a prototype, the more we have to ask participants to pretend and imagine.

We sometimes forget that wireframes and low-fidelity prototypes can look very foreign to those who are not used to seeing them, especially those that still contain lorem ipsum text. Inevitably, there’s at least one participant who asks, “Why is this in Spanish?” Even worse, when a designer or developer has used humorous names or unbelievable data, participants tend to latch onto these kinds of details, taking them out of the reality of the situation.

Low-fidelity prototypes require more interaction between a facilitator and participants, which interrupts the normal flow of tasks. The facilitator often has to explain how the actual software would respond to participants’ actions. Because many screens and actions are missing, the facilitator has to ask participants what they think would happen in response to their actions. This can yield some good insights, but it’s far from the experience people would have with a fully functional user interface. Some participants have no problem imagining and working with a low-fidelity prototype, while others have difficulty getting past the unfinished aspects.

Thinking Aloud

Having participants think aloud while performing tasks is a great way to get insight into their experience, but it’s not what people normally do when they are alone. (If it is, they may need professional help.) Some participants are able to express their thoughts and experiences quite naturally, while others find it difficult and uncomfortable. To deal with those who have difficulty thinking aloud, our training tells us to continually remind participants to think aloud or ask “What are you thinking now?” However, too many prompts can cause further anxiety and disrupt a task flow. Even those who are naturals at thinking aloud have difficulty doing it while performing difficult, cognitively complex tasks.

Another problem is that thinking aloud affects task performance. Because this makes people think more deliberately about what they are doing with a product or service, it can change how people perform tasks, what they notice in a user interface, and how long it takes them to perform a task. For example, it’s common for participants to stop mid-task and discuss something at length with a facilitator.

The Facilitator

Facilitators play dual roles. To build rapport with participants and make them feel comfortable, facilitators must be friendly, reassuring, and empathetic. On the other hand, to avoid biasing participants, they have to maintain a neutral demeanor and be very careful about what they say. Balancing these two roles is very tricky.

The Facilitator’s Neutral Demeanor

In maintaining the required neutral demeanor, facilitators can sometimes seem distant and cold. It’s very difficult to be reassuring and put participants at ease when you cannot even tell them whether they have performed tasks correctly. Facilitators ask participants many questions, but cannot answer very many of of a participants’ questions. It’s no wonder they often feel as though we’re testing them, despite our reassurances to the contrary.

Being Observed

A well-known principle in social psychology is that people change their behavior when they are aware that they are being studied. This unavoidable bias is known as the Hawthorne Effect, from the early 20th century studies of Elton Mayo. [1] Observation affects participants’ performance, and they may change their behavior to fit what they think we expect. We can only do our best to take these biases into account and minimize their effects whenever possible.

The Presence of the Facilitator

Having a facilitator looking over your shoulder, noting your every action, is a constant reminder that you are under observation. Plus, our notetaking can inadvertently give a perceptive participant a sense of what we value. Furious notetaking indicates a participant has just said or done something that is very important to us. No notetaking means a participant has not said or done anything of value to us. This feedback can subtly affect participant behavior, especially for those who want to please a facilitator by providing useful comments.

Because the presence of a facilitator can be disruptive, participants in some usability tests complete their tasks alone in a lab, with the facilitator observing from another room. This removes the discomfort of a facilitator’s observing them directly, as well as unintended feedback from notetaking. But it can be even more impersonal and intimidating when a facilitator communicates with a participant from another room through a speaker—seemingly as the voice of God.

The Observers

For most participants, being observed is the most anxiety-producing and unnatural aspect of user research. While observers in the same room can be disruptive, the idea of unseen observers can produce even more anxiety. The two-way mirrors that are typical in traditional usability labs and focus-group facilities immediately create an intimidating environment and are a constant reminder of the presence of unseen observers. These mirrors can create self-consciousness in people who don’t like to think about how they look and are a distraction for those who enjoy looking at themselves in mirrors.

Who the observers are can also affect participant behavior and comments. Some people are reluctant to provide negative comments when they know the people who created a product or service are observing them. Employees may wonder whether their peers or managers are observing them and may fear that you’re evaluating their job performance and competence.

Recording Sessions

The presence of audio, video, eyetracking, and other types of recording technology can add another layer of anxiety to user research sessions. Unless recording devices are very unobtrusive, their presence is a constant reminder to participants that everything they’re saying and doing is being recorded.

What Will Happen to the Recordings?

Despite our reassurances about security and confidentiality, participants often wonder who exactly might see and hear the recordings. Twenty years ago, when recordings were on video and audio tape, there was a physical limitation on who could view them. These days, with digital video and the Internet, there are no longer any physical limitations to where a video could turn up or who could see it. Might it end up on YouTube? Is a researcher streaming it live to observers around the world? Sure, the recordings may be secure now, but where could they end up in the future?

Could User Research Be More Natural?

With all of these issues, are there ways to make user research seem more natural? Of course, there are. In Part II, I’ll discuss some ways to minimize or eliminate many of the unnatural aspects of user research, including the following:

  • Be a participant yourself.
  • Recruit real people.
  • Minimize the formality.
  • Set the right expectations.
  • Use think aloud when appropriate.
  • Use realistic tasks.
  • Properly explain prototypes.
  • Minimize the effects of observers.
  • Minimize the effects of technology.
  • Conduct more field studies.
  • Use remote research techniques.
  • Do natural observation. 


[1] Shuttleworth, Martyn. “Hawthorne Effect.” Experiment Resources, 2009. Retrieved March 22, 2011.

Principal User Experience Architect at Infragistics

Cranbury, New Jersey, USA

Jim RossJim has spent most of the 21st Century researching and designing intuitive and satisfying user experiences. As a UX consultant, he has worked on Web sites, mobile apps, intranets, Web applications, software, and business applications for financial, pharmaceutical, medical, entertainment, retail, technology, and government clients. He has a Masters of Science degree in Human-Computer Interaction from DePaul University.  Read More

Other Columns by Jim Ross

Other Articles on Usability Testing

New on UXmatters