Capturing User Research
Published: April 2, 2012
“It’s like drinking from a fire hose with a straw, with another smaller straw inside it.” That’s how a colleague who was not allowed to record participants in user research sessions described doing contextual inquiries for a very complex financial application. He was trying to prepare me for my own contextual inquiries with the same client, who didn’t allow any recording devices: No video. No audio. No pictures. Handwritten notes only.
As a consequence, working in an unfamiliar domain, we would be observing very complicated work involving many different applications, documents, and interactions with other employees. Simply trying to observe and understand what was going on would be difficult enough. But trying to simultaneously record all of that information in only handwritten notes, without introducing any delays in the sessions, would be an extreme challenge.
Fortunately, we convinced our client to do the recordings themselves, so they could keep the recordings in their possession at all times. They agreed to allow us to listen to the recordings later in their offices. This eased their security concerns and made our research much easier. It allowed us to focus on what we were observing in the moment and rely on listening to the recordings later to capture detailed notes.
The Difficulty of Capturing User Research
You have to juggle a lot during any user research session, even without the added job of capturing the experience for later analysis. Throughout each user research session, you’re setting participants’ expectations, guiding them to demonstrate tasks, keeping them on track, reminding them to think aloud, managing the pace of the session, and keeping an eye on the clock. You’re also tracking your own behavior, maintaining appropriate eye contact and body language, ensuring you use a neutral tone to avoid bias, and using your interpersonal skills to make participants feel at ease.
Throughout this process, you’re also trying to understand what you’re learning, assessing whether you’ve received the information you need, determining what else you need to cover, and deciding where to take the session next. In the back of your mind, you’re keeping track of the questions and topics you had planned to cover, while remaining flexible and responsive to the natural flow of participant feedback.
Methods of Capturing Information During User Research
As if all of those competing demands on your attention weren’t enough, you also have to capture the information you receive from participants in enough detail to make sense of it later. There are various methods of doing this that you can use, each with its own advantages and disadvantages.
Taking Handwritten Notes
Handwritten notes are the most basic and low-tech method of capturing user research. Note-taking comes naturally to most people, and even when you’re recording sessions, it’s difficult not to take at least some notes.
However, things often happen too quickly during a research session for you to take accurate, complete notes that make sense when you read them later. Trying to write down everything would add to the already high cognitive burden of observing and managing the session. It’s extremely difficult to focus your attention equally on what you’re observing and taking detailed notes.
The Effect of Note-taking on Participants
Doing too much note-taking can affect the interpersonal dynamic between you, the facilitator of a session, and the participant. It’s difficult to maintain eye contact and encourage conversation when you’re focusing too much attention on capturing notes about the session. Note-taking can also contribute to a participant’s feeling of being tested or being the subject in an experiment.
Another important impact of note-taking is that it can bias participant behavior. Sudden, furious scribbling indicates to participants that they’ve just said something profound, while no note-taking implies that you’ve lost interest in what they have to say. Subtle cues like this can lead participants to try to please you by saying things they think you want to hear.
Note-Taking in Remote Sessions
It’s much easier to take notes when doing remote user research. Because participants are not in the same room with you, you don’t have to worry about eye contact, body language, or how your note-taking affects participants. This relieves part of your cognitive burden, enabling you to focus your attention on your computer screen and your notes.
Note-Taking in Structured Research
It’s much easier to take notes during structured research activities than more free-flowing or open-ended ones. For instance, usability testing has a narrow scope, uses predefined tasks and questions, and proceeds in a predictable order, making it very easy to take notes. You can add note-taking shortcuts to your discussion guide in advance of your research sessions by creating specific areas for quickly noting task successes or failures, common actions, errors, and other expected events. You can paraphrase especially interesting comments or describe key actions, and put a flag and timestamp next to them as a reminder to view them later in the recordings.
Note-Taking During Less-Structured Research
Other types of user research activities such as interviews, contextual inquiries, and ethnography are much less structured and predictable, which makes note-taking especially difficult. Unlike usability testing studies, you often don’t know exactly what you’ll be observing and talking about until you’re in the moment. Such sessions require a lot more thinking on your feet to understand what you’re seeing, ask the appropriate questions, and move the session in the appropriate direction. Since there’s no time to decide what is important for you to capture, it’s tempting to try to capture everything, then sort out what’s important for you to retain later.
Considering all of the difficulties of taking handwritten notes, it’s best for you to use some additional method of capturing your research.
For many people, typing notes on a notebook computer is certainly faster and easier than writing them by hand. But doing this in the same room with user research participants can be distracting and seem impersonal or even rude. While a tablet might be a less intrusive note-taking device than a notebook computer, most people find it too difficult to type quickly and accurately on a tablet. Adding a keyboard makes it easier to type on a tablet, but then you face the same problems as with a notebook computer. I’d like to have a tablet on which I could write notes using a stylus. That would be a great way to save paper and easily digitize my notes.
Involving a Separate Note-Taker
Splitting note-taking and facilitating between two people makes the people in both roles more effective. The note-taker can focus entirely on observing, listening, and documenting the situation, while the facilitator can focus on actively listening and observing the participant to better facilitate the session in the moment.
Of course, you need to have someone you can trust to take detailed and accurate notes. It’s important to choose someone who is observant, knows what is important to focus on, and can write quickly. Since it’s usually difficult to understand someone else’s notes, it’s best to have that person type them up afterward, filling in the additional detail that is necessary to make them understandable to others.
There’s no way of capturing user research sessions more comprehensively than to record them, then listen to the recordings later and type up your notes from the recordings. Recording audio is especially useful for more unstructured and unpredictable methods of user research such as interviews and contextual inquiries, for which note-taking is more difficult. If you know you’ll have time to listen to the recordings later, you can minimize your note-taking, allowing you to focus on the sessions.
Listening to the recordings is especially important when you’re dealing with complex subject matter. I often find that I don’t completely understand what I’ve observed until I go through the sessions again by listening to the audio recordings. Hearing things a second time reveals what you missed the during the sessions and gives you a deeper understanding of your findings.
And regardless of whether you take handwritten notes, it’s always good to have audio recordings as a backup. Even when you won’t have time to listen to an entire set of recordings, it’s nice to have them available in case you want to go back and review particular parts of the sessions.
You should record video whenever it’s important for you to be able to review the visual elements of your user research sessions. For example, in usability testing, contextual inquiries, and ethnographic observations, it’s important to see participants’ actions. For other research activities such as interviews, the visual element is less important, so audio recordings are sufficient.
Video also gives you a powerful way of allowing others to experience your user research. Although you can report your findings in words, it’s often much more impactful for your audience to see and hear research participants by watching video clips.
So, if video provides the benefit of capturing both the audio and visual elements of research sessions, why not capture video all the time? The main problem with recording video is that it’s much more intrusive than recording audio. Many people feel uncomfortable and self-conscious at the thought of being on camera. Because video cameras must point directly at participants, they can be very intrusive, constantly reminding participants that you’re recording everything they’re saying and doing. By contrast, audio recorders are much less intrusive, both because of their small size and because you can place them off to the side.
It’s important to weigh the advantage of capturing the visual element against the disadvantage of making participants uncomfortable. Audio is often the best option if you can use your notes to capture the visual elements that your audio recordings don’t provide.
Taking photos is an effective way to capture both the research environment and the tools participants use. Photos serve as a reminder of the things you’ve observed during research sessions and provide a good way of illustrating your research findings for others. If you record only audio, you can use photos to supplement the visual aspects that you’re recordings didn’t capture.
While being photographed feels intimidating to some participants, taking snapshots with your smartphone camera is a less obtrusive way of capturing photos. Although the photo quality may not be as good as that you’d get with a digital camera, a smartphone seems more informal and less like a camera.�If you turn off the shutter sound effect and the flash, smartphones can be really subtle. This adds up to a less alienating experience for participants and some very useful documentation for you and anyone with whom you need to share your research findings.
Using Event-Logging Software
A more advanced alternative to including a second note-taker on your research team is having another person log events during research sessions. Logging software lets you place markers and notes at specific moments in a video recording. When watching logged videos later, you can use the markers to quickly jump to specific points of interest. That makes it much easier to find specific parts of sessions to review the information and include it in video clips.
A disadvantage of this approach is that you need an additional person to do the logging. You may not have the luxury of including another person who can attend all of your research sessions on every project. However, logging is a good job for either an intern or an entry-level user researcher who is a less expensive resource and can benefit from the learning experience. So consider the possibility that there might be affordable ways of getting someone to fill this role.
Using a Livescribe Smartpen
While logging links your notes with specific moments in a video recording, Livescribe Smartpens link written notes with specific moments in audio recordings. Smartpens combine the best elements of handwritten notes, audio recordings, and logging. Unlike the logging setup you might use in a usability lab, Smartpens are portable, so you can use them in any research situation. When you use a Smartpen with a special notebook, it records audio as you write your notes. Afterward, you can tap your notes anywhere with the Smartpen to hear the corresponding moment in the recording. You can also download a pencast PDF file that combines your written notes with the audio recording.
Relying on Your Memory
Although you use can various methods to capture your research findings, your brain first captures much of the information, where it remains in your memory. Notes and recordings are tools that you can use to refresh memories of your research sessions, but they don’t substitute for your actually experiencing the sessions in the first place. If you’ve ever tried to read others’ notes or review their recordings, you know that doesn’t give you the same understanding as first-hand observation of the research sessions would have.
A good technique for accessing your memories is to wait a day or two after your research ends. Then go through your research discussion guide and type up the main things you remember, using its tasks and questions to guide you. You’ll find that the overall themes naturally come to the surface. If time allows, you can then go through your notes or listen to the recordings to add in the finer details.
Considerations in Choosing a Method of Capturing Information
How can you choose between these different information-capture methods? Though the best methods may vary from study to study, consider the following issues.
The Subject Matter of Your Research
How complex is the subject matter? If it’s something you can easily understand during your research sessions, taking notes and relying on your memory may be enough. However, if it’s something more complicated, you may need to record and review the sessions in detail to fully understand the material.
The Types of Information You Need to Capture
Will the sessions involve only discussion, or is there a visual element that you need to observe? If it’s important for you to see the visual elements, you should probably record video. If it’s not important to review the visual elements, audio recordings and notes may be enough.
What Research Method You’re Following
Will your research follow a structured set of tasks and questions, as in usability testing, or will it be more loosely structured and unpredictable, as in a contextual inquiry? The more structured and predictable your research sessions, the easier it is to capture everything in your notes. Recording audio or video becomes more necessary when sessions are loosely structured and unpredictable.
How Much Repetition Is Likely
How much repetition will there be in the tasks you observe? If you’ll see and hear the same things over and over again, the major themes will stick in your memory. You won’t have to rely on your notes or recordings as much. With numerous chances to capture the same information, taking notes may be sufficient.
The Effect of Your Method on Participants
What effect will your information-capture method have on participants? If you’re concerned that recording video might make participants uncomfortable or that taking notes might get in the way of your connecting with participants, you may want to record audio.
What Your Audience Needs
Are you capturing this information for only your own use, or are there others who will need to review the data that you gather? While you may be able to understand your own notes, others may need video or audio clips to understand your findings.
How Much Detail You Need
How detailed do your findings need to be? If you’re reporting just on the main issues, you don’t have to capture as much detail as you would if you need to document all of the steps in a complex process.
How Much Time You Have
How much time will you have to do your analysis and report on your findings? If you need to do a quick analysis, then immediately report your findings, you won’t have time to review recordings or even do a detailed review of your notes. In that case, brief notes are more appropriate.
The Future of Research Information Capture
It’s interesting to think of what the future might bring in information-capture technology for user research. In my dreams, an ideal tool would be on a tablet, reducing the massive amount of paper that I currently waste when capturing handwritten notes. It would allow me to view a discussion guide and add handwritten notes using a stylus. My notes would be synced with either an audio recording or a wireless video recording, which would make it easy to jump to any point in a recording that corresponds to particular notes. The application would then take my handwritten notes and automatically convert them to text that I could manipulate in a word processor. Do you know of any tools that would let me achieve this? If not, I can dream. In the meantime, I’ll be taking plenty of handwritten notes on paper and backing them up with audio or video recordings.