Becoming a Spy: Covert Naturalistic Observation

Practical Usability

Moving toward a more usable world

A column by Jim Ross

November 1, 2013

As I mentioned in a previous column, user research is unnatural. When we conduct a user experience study, we can’t help but introduce unnaturalness. Following ethical research guidelines, we inform participants about the study and get their consent to participate, but when people know we’re observing them, their behavior changes. This is known as the Hawthorne Effect. [1]

Most researchers are aware of these limitations of research, but after a while, we tend to take them for granted and may forget how strange it can feel for a participant to be in a usability lab. We may try to overcome this artificiality by going into the field and observing people in their natural environments, which certainly is more natural than conducting research in a lab. But knowing that we’re observing them can still be an uncomfortable experience for participants, and we may forget that our presence affects their behavior.

Champion Advertisement

Covert Naturalistic Observation

So, how can we observe natural behavior if our mere presence affects what people do? Don’t tell them that you’re observing them. At this point, you may be thinking: Wait a minute. Isn’t that unethical? I’ll get to that, so please read on.

This type of study is known in psychology, anthropology, and other social sciences as covert naturalistic observation. It is the opposite of the techniques we typically use, which are forms of overt naturalistic observation. Being covert means observing behaviors in their natural contexts without any intervention or influence by the researcher and without participants knowing that they’re being observed.

Covert naturalistic observation isn’t a completely new idea. In addition to its use in the social sciences, it’s practiced by law enforcement agencies, spies, and amateur people-watchers all over the world. But for some reason, its use in user experience research is somewhat rare.

What Are Its Advantages?

By ensuring that participants are unaware that we’re observing them, we eliminate any effect that we have on participant behavior—the Hawthorne Effect. We can see what people really do rather than what they say they do or what they show us when they know we’re studying them. What better way to see natural behavior than to go out and observe what is really happening outside a study?

What Are Its Disadvantages?

Being covert requires you to hide the fact that you’re observing, which makes it more difficult to observe and take notes. So it’s not very useful for studying interactions up close—such as making detailed observations of a person using a user interface. It’s more suited for observing actions and behaviors that you can see from a distance.

Being covert eliminates your ability to ask questions or discuss what you’ve observed with participants. So it can leave you with additional questions and require you to make assumptions.

In the social sciences, some have criticized covert research as being ethically unsound, deceptive, and an invasion of privacy. [2] Without the knowledge that they are being observed, participants can’t provide their informed consent, and they have no choice about whether to participate. However, despite these objections, covert naturalistic observation has its advocates who argue that, in the right situations, there are ways to do this research ethically. I’ll discuss those later in this column.

When Should You Use Covert Naturalistic Observation?

The best application of covert naturalistic observation is for studying the behavior and interactions of groups of people in public places. So it’s ideally suited for research projects for service design, workflow redesign, and process redesign. Some examples of questions to which covert research could provide answers include the following:

How can we speed up the process and improve the passenger experience of airport security lines?
How can we make the emergency-room waiting area a better experience for patients and those who accompany them?
How can we improve customers’ shopping experience and increase sales in our clothing store?

As I mentioned earlier, being covert is difficult when you need to see details close up—such as how people use a user interface. Overt methods such as contextual inquiry are better suited for those types of studies.

Tips on Covert Naturalistic Observation

Like most user research methods, covert naturalistic observation is more complex than it may seem at first glance. Just as overt observation is more than just watching people do things and asking questions about what they do, covert observation is more than just people-watching. The following tips will help you to get the most out of this method of research.

Gather Intelligence

Before you go out to observe, learn as much as you can about the domain that you’re studying, the characteristics of user groups, and the behaviors and tasks you’ll be observing. Review any previous research to which you have access. By being informed, you’ll know more about what to look for, and you’ll be better able to make sense of what you see.

Plan What You Want to Observe

If, for example, you wanted to study what people do at the airport while waiting for a flight, it would be a mistake to think, I’ll just go to the airport and see what people do. Without planning in advance what your study will focus on, you would quickly become overwhelmed by the amount of information you’d take in. This makes it especially important for you to plan what you want to learn, the questions you want to answer, and how you’re going to gather the information you need. Knowing what to look for makes it easier to focus your observation on what’s important and discard the rest.

Do Some Reconnaissance of the Location

As part of your planning, scout out the location ahead of time. Get answers to the following questions to determine where and when to observe people and how to appear natural and remain unnoticed.

What’s the layout like?
Where are the best places from which to observe?
What’s the activity level like at certain times of the day?
When is the best time to observe the behavior you want to see?
What are people doing and what are they wearing?

If appropriate, notify the management or security of your research. For example, if you’re observing people in an airport, the customers might not notice you, but security might. It’s better to let them know what you’re doing in advance.

Observe as a Group

There are advantages to having two or more people observe as a group:

Each person can observe behavior in a different location and focus on different people.
Each person can focus on observing different behaviors or answering different research questions.
You’re less likely to miss something.
You have someone to discuss your observations with.

If you work in a group, be sure to coordinate where each person will be, what each person will do, and what each person should focus on.

Observe During Several Shorter Sessions

Instead of one long observation session, it’s usually better to break your research into several shorter observation sessions. Doing shorter sessions has the following advantages:

You can see what happens over several sessions—perhaps at different times of the day, observing different people, and observing different situations.
You can see whether there are repetitive behaviors, situations, and patterns that recur over several sessions.
It’s easier to remain mentally sharp, observant, and avoid fatigue.
You’re less likely to be overwhelmed with too much information, so it’s easier to remember what you observed.
You’d appear more natural and would be less likely to get caught if you didn’t hang around too long.

Blend In

To remain covert, you must blend in and can’t appear to be observing people. That’s easier in some situations than others, but in most situations, the following tips will help you to blend in:

Dress the same way everyone else does.
Do what others are doing.
Keep your level of eye contact normal.
Maintain a relaxed posture.
Use a smartphone or tablet as cover—to appear like you’re doing something, not paying attention to your surroundings. You can pretend to talk on the phone or pretend to be texting, while you’re actually observing and taking notes.

Observe Effectively

Observation sounds deceptively simple. You just watch people, right? Actually, it takes practice to observe effectively in an unstructured situation. Over time, you’ll improve your observation skills and will be able to focus on what’s important. Some things to look out for and note include the following:

repeated behaviors and patterns
unique behaviors
body language
the way people interact with other people, objects, and the environment
the order of people’s actions
similarities and differences between people

Don’t Be Obvious About Taking Notes

Remaining covert will prevent your taking notes as easily, obviously, or extensively as you could in an overt observation such as a contextual inquiry. So don’t even try to take detailed notes. Instead, rely more on your memory. Take very brief notes to capture major themes or observations that will trigger your memory later on. When the observation is over, you can type up more extensive notes from your memory, expanding on your brief notes.

As I mentioned earlier, smartphones and tablets are great cover for taking notes without your looking like you’re taking notes. To anyone who notices you, it looks like you could be using an app, texting, or writing an email message. If you must take notes on paper, make it look like you’re reading something and writing notes about what you’re reading.

Don’t Worry About Recording

You’re primarily there to observe, not record the situation. So focus primarily on observing, then if necessary, capture some photos or video at the end of the session. The purpose of these photos or video is not to record the entire observation. Their purpose is to capture important things, help you to remember what you observed later on, and show those who didn’t accompany you on the observations what you’ve observed.

To remain covert, you’ll need to take photos and record video very surreptitiously. That’s another reason to wait until the end of your sessions to take photos. If you get caught, you’ve already finished observing, so it doesn’t really matter.

Smartphone cameras are ideal for this sort of research because they’re not as obvious as traditional still or video cameras. If you turn off the flash and shutter sound, people often won’t realize you’re taking pictures. Don’t record audio. It’s not necessary, and recording audio without consent is illegal in most places.

Do a Debrief Afterward

Immediately after each observation session, sit down and type up your detailed notes, while the information is still fresh in your mind. First, capture whatever’s at the top of your mind—such as your overall observations, patterns, themes, conclusions, and any further questions that you may have. Then review the brief notes that you took during the session to see whether those spark any additional memories of things that you should note. You may also want to draw a map or a layout of the physical space.

Once each individual observer types up his or her notes, get together with the other observers to compare your notes and discuss what each of you observed. Look for commonalities and differences. Have one person take notes on this discussion. At the end of all of your observation sessions, combine the notes from each observer, and do your analysis.

Do Covert Research Ethically

Covert research can be ethical if you’re careful not to violate any laws or ethical boundaries. Conduct covert observation only in public places, where passersby might reasonably observe the same situation. Consider whether there’s an expectation of privacy in the situation or location. For example, it may be ethical to observe customers shopping in a clothing store, but it’s certainly not okay to observe them in a dressing room.

Focus on people and situations in general, not specific individuals. Unlike a spy or police officers on a stakeout, you’re not targeting, following, or stalking a specific person. You’re situated in a public place observing what people in general do.

Be careful to protect people’s privacy by not revealing their identities in photos or videos or their names in reports. Block out their faces unless you’ve received consent to show them.

Don’t deceive participants. You’ll try to blend in, so people won’t notice you, but you’re not going under cover. If you get caught, you should just be honest about what you’re doing. Many people won’t mind and might even find it interesting.

Combine Covert Research with Other Methods

Because covert naturalistic observation doesn’t allow you to ask questions or discuss what you observe with participants, it can bring up a lot of questions and require you to make assumptions. To answer those questions and test your assumptions, it’s useful to combine covert research with more traditional overt methods such as user interviews or contextual inquiries. You can also use traditional, overt methods first, so you’ll have a better understanding of the subject matter when you do your covert observation.

Comparing your covert and overt findings can be an interesting exercise. You can determine whether there were any interesting differences between what people did when they knew you were observing them versus when they didn’t know you were observing them.

Should You Do Covert or Overt Research?

Covert or overt observation, which is better? It depends on the situation. Both methods have advantages and disadvantages. Overt observation affects participants’ behavior. Being covert eliminates this problem, but limits what you can observe and prevents your asking questions. But as long as you’re aware of these limitations and try to minimize them, you can choose overt and covert methods as appropriate to your research and what would best enable you to understand the participants’ experience.

References

[1] Shuttleworth, Martyn. “Hawthorne Effect.” Explorable.com, 2009. Retrieved October 10, 2013.

[2] Lugosi, Peter. “Between Overt and Covert Research: Concealment and Disclosure in an Ethnographic Study of Commercial Hospitality.” Qualitative Inquiry, 2006. Retrieved October 10, 2013.

In User Research

Norman Boyd

November 6, 2013 12:58 AM

It occurs to me that any good customer-service person should be doing such observation all the time. As conscientious service personnel, we look to interrupt at points on a customer’s journey where we see accidents and work to correct them. I’d like to see a bit more on ethics here because the example given, of a dressing room, is rather extreme. However, this is a great article.