Conducting Large-Scale User Research

Practical Usability

Moving toward a more usable world

A column by Jim Ross
September 2, 2014

What would you do if you were asked to do an extremely large-scale user research project? What do I mean by large? How about performing more than 150 contextual inquiries? How would you handle such a large amount of information from many different user groups, whose subject matter covers such a large scope? Doing unmoderated research such as online card sorting and unmoderated usability testing is an easy way to get a large number of participants, but what if you need to do moderated sessions?

Admittedly, needing to do such large-scale research is a rare situation. UX professionals usually face the opposite problem—not having enough participants. That’s why, when you’re suddenly faced with a large-scale research project, it can seem so intimidating or even overwhelming.

Champion Advertisement
Continue Reading…

Why Would You Need So Many Participants?

First, you might ask, why would you ever need to have that many participants? But when you’re studying a broad subject area that has many different user groups and encompasses many different tasks, and you need to include enough people from each user group, you’ll inevitably end up with a large number of participants.

For example, I once conducted user research to understand the problems with a large corporation’s implementation of SAP’s Human Resources software. Many different types of employees used this software for entirely different business purposes. The user groups included recruiters, HR generalists, managers, employees, payroll staff, trainers, staffing personnel, and several other groups. Ideally, we wanted to do research with at least five employees from each user group, so we conducted over 50 two-hour contextual inquiries.

On another project, we needed to understand eleven business processes across two merged companies. Each business process involved many different roles who performed a variety of tasks. Even when we skipped some less essential user groups and limited the research to only two or three participants per group, we still ended up with over 150 participants.

What’s Difficult About Doing Large-Scale User Research?

With everything taking longer and being on a larger scale, more complex, and more expensive, a large-scale user research project leads to larger problems such as the following:

  • difficulties getting up to speed on subject matter with such a wide scope
  • needing to find, recruit, schedule, and coordinate so many participants
  • the need for extensive travel and the issues of coordination, scheduling, expense, and fatigue that involves
  • risks regarding what can happen over a long period of time—such as key people leaving the project, information becoming out of date, and the potential for forgetting the findings from early research
  • difficulties analyzing such a huge amount of qualitative data
  • challenges of presenting findings and recommendations whose scope is so great, without overwhelming your audience

How to Conduct Large-Scale User Research

By following these tips, you’ll be able to conduct a large-scale user research project successfully.

Manage the Project Scope

First, ask the question, “Does the research project scope really need to be this large?” If not, try to narrow its focus. If its scope does need to be so large, determine whether it’s possible to break the project into more manageable phases.

Reduce the Scope

Since the number of participants drives a research project’s time and cost, it might seem that the easiest way to reduce the project’s scope would be simply to reduce the number of participants. But, unless you also reduce the scope of the subject matter that you’re covering, that would be the worst way to reduce scope. You’d end up either having fewer participants per user group or skipping some groups, which would result in flimsy, superficial, unreliable research results that you’d base on too few observations.

It’s much better to focus your research by either reducing the scope of the subject matter or breaking your research into phases, focusing each phase on a different subject area or user group.

Start with a Small Scoping Project

If you’ve ever scoped a user research project, you’ve probably run into the catch-22 of having to determine the number of participants you’ll need before you know enough about the subject matter to know how many different user groups there are. While an in-depth conversation with your clients can provide enough information to scope a smaller project, for large projects, it makes sense to begin with a scoping project.

On a scoping project, you may conduct interviews or workshops with key stakeholders to better understand the subject matter, the user groups, and their tasks. The goal is to gather enough information to accurately scope the project. The research that you do during the scoping project reduces the amount of up-front research that you’ll need to conduct for the actual research project, so your effort won’t go to waste.

Determine Who Will Do the Research

On a small project, one team typically conducts all of the research sessions, but when there are many participants, you may want save time by having multiple teams conduct the research.

Break into Teams?

Using multiple teams shortens the duration of the research. For example, each of two research teams could meet with 50 individual participants in half the time it would take one research team to meet with 100 participants. The overall hours of research remain the same, but the duration is cut in half. Adding more research teams reduces the duration even further.

The disadvantage of multiple teams it that it divides the knowledge that you get from the research between multiple teams. Instead of one team gaining a deep understanding of the research by observing and hearing everything firsthand, each person becomes an expert in only the findings from sessions that he or she observed firsthand. Although the teams certainly can do their best to share the information, it’s not the same as seeing it firsthand.

If you do divide your research between teams, divide it logically, so each team covers a particular subject area or user group. Then each team gains an expert understanding of their own subject area.

Beware of Turnover

While it may be tempting to save money by having only one person on each research team, the longer the project, the greater the possibility that researchers might leave before the end of the project. Because so much of the information from research resides in a researcher’s head, it’s not easy to transfer information to someone new. If a team of two people conducts research, there’s a greater likelihood that at least one person who has first-hand knowledge of the research sessions will remain on your team.

Determine the Order of the Research

When you’re researching a process or subject matter that’s complex, you would ideally want to observe that process in the order that it happens. So you would first observe the people who perform the first part of the process, followed by the people who perform the second part of the process, and so on.

However, that’s easier said than done when you’re trying to coordinate the schedules for tens or hundreds of participants in many different locations. Travel logistics, time constraints, and expense also make it difficult to visit participants in an ideal order. Instead, you’ll often settle for scheduling participants around their availability and travel logistics.

When you have to meet with participants in a haphazard order, it helps to build a model of the overall process, so you can keep track of how things happen in the overall process and how the participants relate to each other. Conduct stakeholder workshops before the user research sessions to get your initial understanding of the users, their tasks, and the subject matter. By obtaining that information first, you’ll be able to build your initial model to provide context as you meet with each participant.

Coordinate the Efforts Between Teams

If you have multiple research teams, it’s very important to coordinate their efforts. Conduct daily debriefing sessions to keep the research teams in sync. Even brief, 30-minute phone calls can be helpful in keeping physically separate teams close. These sessions allow teams to share and compare their findings, observe trends, propose changes to the research process, and feel connected to one another.

Keep Up with the Data

A typical user research project generates and collects a lot of notes, recordings, photos, files, and physical artifacts. In large-scale research, this flood of data multiplies to a level that can be overwhelming. How do you keep track of all this data and keep what you’ve learned in mind?

Record Sessions, but Don’t Depend Too Much on Your Recordings

You can’t capture everything in written notes, and you can’t possibly remember every detail, so recording audio and video is essential. When you have time to review the recordings and take additional notes from them, this deeply instills the information in your mind. Doing that results in your experiencing every session at least twice—first in person, then through the recording—and taking notes lets you think more deeply about what you’re observing. However, it’s extremely time consuming to review each recording in full—typically taking at least two to three hours to view every hour of the recording.

An alternative is to have one person take very detailed notes during the sessions and review only the parts of a recording where your notes need clarification. Another alternative is to send the recordings to a transcription service, which will type up notes for you. However, this can be expensive, and the information won’t be as deeply ingrained into your mind as if you type notes yourself, while reviewing the recordings.

Regularly Type Up Your Notes

Regardless of whether you’re going to review the recordings, on a large-scale research project, it’s extremely important to type up your written notes at the end of each day. The last thing you should do is wait until the end of all your research, then finally deal with a huge stack of notebooks from sessions that took place weeks ago.

I don’t know about you, but my paper notes from each research session are a disorganized list of paraphrased comments, participant actions, and observations. When I read them shortly after the session, I can understand what I meant, and I can remember what I saw. Then, as I type up my notes, I can add more detail and context than I captured in the words I wrote down during the session. But as days pass and I see more and more participants, my memories begin to fade. This problem compounds when there are tens or hundreds of participants and the research takes place over many weeks or months. If you don’t stay on top of typing up your notes, you’ll forget a lot of details and quickly become overwhelmed.

Use a Survey Tool to Unify Everyone’s Notes

Using an online survey tool is a good way to organize and unify notes from multiple research teams. Last year, when I conducted a large-scale research project with over 150 contextual inquiries, we used Wufoo, an online survey tool, to coordinate the notes from two research teams. This gave us a central and safe location for entering our notes instead of risking losing our physical notebooks. Although each team had its own style of note taking, the standard questions on the form gave our notes consistency and reminded us of things that we needed to capture.

Because we used an online survey tool, we could look at the other team’s notes to see what kinds of things they were finding in their research. Everyone could see how often we were updating the survey with our notes, which encouraged us to update them at the end of each day. As the data accumulated, the standard format made it easy to compare specific topics or issues across participants. We could even run reports to find specific issues.

Manage the Data

Large-scale research generates a ton of paper notes, electronic documents, photos, audio recordings, videos, and paper documents that you’ve gathered from participants. These initially reside on multiple computers, tablets, phones, cameras, and other devices belonging to various people. If you don’t have a well-organized method for naming and storing all of this data, you can end up with a confusing mess. Determine where you’re going to store these files, how you’re going to organize them—by participant, by topic, by issue, or something else—and use a file-naming convention.

Analyze the Data

The voluminous amount of data that multiple teams have gathered obviously poses a challenge when it comes to analysis. If you’ve used a survey tool to capture everyone’s notes throughout the project and all of your files are well organized, you’ll be in a good position to start your analysis.

Provide Time for Individual Analysis First

Because multiple people have conducted the research, it makes sense to analyze the data as a group, but first, let people do their own initial analysis of the information on their own. Many people first need time alone to think about what they’ve observed before discussing it with others.

Analyze the Data as a Group

Next, conduct group analysis sessions to organize and discuss the information that you’ve gathered. Doing affinity diagrams is a good way for multiple groups to combine their findings, putting each finding on an individual Post-it note, then organizing the notes into groups to reveal common themes. Mapping out processes and tasks using Post-it notes is also a good group activity because it lets the group visualize the tasks and come to a common understanding of them. Along with the tasks, group members can use different colored Post-it notes to denote problems in the process, the use of specific tools or technologies, and environmental factors.

Keep the Client and Project Team Updated

Obviously, you need a lot of time to analyze large-scale research data. The problem is that, by the time analysis begins, the research itself has already taken up a lot of time, so your client and project team may be getting antsy about having the findings. They may not understand what analysis involves or why it takes so long. To keep them satisfied, frequently communicate status during research and analysis. Provide previews of your findings such as anecdotes about what you’ve observed. Just make it clear that you’re not providing fully analyzed or final information.

Deliver the Findings

Once your analysis is complete, you’re left with a mountain of information and the quandary of how to most effectively deliver it. Should you provide a high-level summary or all of the details? If you provide only the details, you’ll likely bore and overwhelm them. If you provide only a high-level summary, they may wonder why they spent all that time and money to get only vague generalities. So you need to provide both a summary and details, but this is usually difficult to achieve in one deliverable.

Create Several Deliverables

The solution is to create several deliverables—a report with the details and a presentation with just the summary information. Your report can provide great detail and serve as a long-lasting archive that people can refer to later to better understand the findings—even if they didn’t attend or don’t remember your presentation. Once you’ve created a detailed report, this frees you up to create an engaging, high-level presentation that can focus just on the most important points.

If your research covers several different topics or has different audiences, it’s often better to create several audience-specific presentations. For example, once I gave an extremely long presentation to representatives of several different departments in a company. Each group was bored and perked up only when I got to the findings and recommendations that pertained to their department. It would have been better to create and deliver several smaller presentations that I’d targeted to each department.

Remain Calm

Conducting large-scale research with hundreds of participants, in an unfamiliar subject area, can be very intimidating. Once, when I was slightly freaked out about undertaking such a complicated project, my manager encouraged me by making an analogy to a marathon. You run a marathon one step at a time. Instead of focusing on the whole thing at once, focus on one step at a time, and eventually, you’ll get there. 

Principal UX Researcher at AnswerLab

Philadelphia, Pennsylvania, USA

Jim RossJim has spent most of the 21st Century researching and designing intuitive and satisfying user experiences. As a UX consultant, he has worked on Web sites, mobile apps, intranets, Web applications, software, and business applications for financial, pharmaceutical, medical, entertainment, retail, technology, and government clients. He has a Masters of Science degree in Human-Computer Interaction from DePaul University.  Read More

Other Columns by Jim Ross

Other Articles on User Research

New on UXmatters