Why Would You Need So Many Participants?
First, you might ask, why would you ever need to have that many participants? But when you’re studying a broad subject area that has many different user groups and encompasses many different tasks, and you need to include enough people from each user group, you’ll inevitably end up with a large number of participants.
For example, I once conducted user research to understand the problems with a large corporation’s implementation of SAP’s Human Resources software. Many different types of employees used this software for entirely different business purposes. The user groups included recruiters, HR generalists, managers, employees, payroll staff, trainers, staffing personnel, and several other groups. Ideally, we wanted to do research with at least five employees from each user group, so we conducted over 50 two-hour contextual inquiries.
On another project, we needed to understand eleven business processes across two merged companies. Each business process involved many different roles who performed a variety of tasks. Even when we skipped some less essential user groups and limited the research to only two or three participants per group, we still ended up with over 150 participants.
What’s Difficult About Doing Large-Scale User Research?
With everything taking longer and being on a larger scale, more complex, and more expensive, a large-scale user research project leads to larger problems such as the following:
- difficulties getting up to speed on subject matter with such a wide scope
- needing to find, recruit, schedule, and coordinate so many participants
- the need for extensive travel and the issues of coordination, scheduling, expense, and fatigue that involves
- risks regarding what can happen over a long period of time—such as key people leaving the project, information becoming out of date, and the potential for forgetting the findings from early research
- difficulties analyzing such a huge amount of qualitative data
- challenges of presenting findings and recommendations whose scope is so great, without overwhelming your audience
How to Conduct Large-Scale User Research
By following these tips, you’ll be able to conduct a large-scale user research project successfully.
Manage the Project Scope
First, ask the question, “Does the research project scope really need to be this large?” If not, try to narrow its focus. If its scope does need to be so large, determine whether it’s possible to break the project into more manageable phases.
Reduce the Scope
Since the number of participants drives a research project’s time and cost, it might seem that the easiest way to reduce the project’s scope would be simply to reduce the number of participants. But, unless you also reduce the scope of the subject matter that you’re covering, that would be the worst way to reduce scope. You’d end up either having fewer participants per user group or skipping some groups, which would result in flimsy, superficial, unreliable research results that you’d base on too few observations.
It’s much better to focus your research by either reducing the scope of the subject matter or breaking your research into phases, focusing each phase on a different subject area or user group.
Start with a Small Scoping Project
If you’ve ever scoped a user research project, you’ve probably run into the catch-22 of having to determine the number of participants you’ll need before you know enough about the subject matter to know how many different user groups there are. While an in-depth conversation with your clients can provide enough information to scope a smaller project, for large projects, it makes sense to begin with a scoping project.
On a scoping project, you may conduct interviews or workshops with key stakeholders to better understand the subject matter, the user groups, and their tasks. The goal is to gather enough information to accurately scope the project. The research that you do during the scoping project reduces the amount of up-front research that you’ll need to conduct for the actual research project, so your effort won’t go to waste.
Determine Who Will Do the Research
On a small project, one team typically conducts all of the research sessions, but when there are many participants, you may want save time by having multiple teams conduct the research.
Break into Teams?
Using multiple teams shortens the duration of the research. For example, each of two research teams could meet with 50 individual participants in half the time it would take one research team to meet with 100 participants. The overall hours of research remain the same, but the duration is cut in half. Adding more research teams reduces the duration even further.
The disadvantage of multiple teams it that it divides the knowledge that you get from the research between multiple teams. Instead of one team gaining a deep understanding of the research by observing and hearing everything firsthand, each person becomes an expert in only the findings from sessions that he or she observed firsthand. Although the teams certainly can do their best to share the information, it’s not the same as seeing it firsthand.
If you do divide your research between teams, divide it logically, so each team covers a particular subject area or user group. Then each team gains an expert understanding of their own subject area.
Beware of Turnover
While it may be tempting to save money by having only one person on each research team, the longer the project, the greater the possibility that researchers might leave before the end of the project. Because so much of the information from research resides in a researcher’s head, it’s not easy to transfer information to someone new. If a team of two people conducts research, there’s a greater likelihood that at least one person who has first-hand knowledge of the research sessions will remain on your team.
Determine the Order of the Research
When you’re researching a process or subject matter that’s complex, you would ideally want to observe that process in the order that it happens. So you would first observe the people who perform the first part of the process, followed by the people who perform the second part of the process, and so on.
However, that’s easier said than done when you’re trying to coordinate the schedules for tens or hundreds of participants in many different locations. Travel logistics, time constraints, and expense also make it difficult to visit participants in an ideal order. Instead, you’ll often settle for scheduling participants around their availability and travel logistics.
When you have to meet with participants in a haphazard order, it helps to build a model of the overall process, so you can keep track of how things happen in the overall process and how the participants relate to each other. Conduct stakeholder workshops before the user research sessions to get your initial understanding of the users, their tasks, and the subject matter. By obtaining that information first, you’ll be able to build your initial model to provide context as you meet with each participant.