Recruiting Participants for Unmoderated, Remote User Research

Practical Usability

Moving toward a more usable world

A column by Jim Ross
September 6, 2010

It seems new, online tools for conducting unmoderated, remote user research emerge every week. While this method of doing user research and these tools have generated a lot of interest and discussion, it is also important to consider how best to recruit participants for unmoderated studies. Though one might assume this would be similar to recruiting for moderated studies, very different methods of recruiting are necessary to find the large number of representative participants unmoderated studies require and convince them to participate.

This column explores the differences between recruiting for moderated and unmoderated user research. It discusses the three primary techniques for recruiting participants for unmoderated, remote user research and helps you decide which technique to choose, based on your study’s needs. For tips on how to recruit participants for moderated user research, please see my previous column, “Recruiting Better Research Participants.”

Champion Advertisement
Continue Reading…

What’s the Difference Between Moderated and Unmoderated User Research?

In moderated studies, the user researcher and participants meet either in person or remotely via phone and Web-conferencing software. Study sessions typically last an hour or more. Traditional usability testing, field studies, user interviews, and focus groups are examples of moderated studies.

In unmoderated, remote research, the user researcher and participants never meet. Participants complete a study at their own location, on their own time. An online tool collects data and presents the results to the researcher. Sessions typically require only a fraction of the time moderated research requires. Examples of unmoderated, remote research are online card-sorting studies, surveys, and online usability testing.

How Is Recruiting for Unmoderated, Remote User Research Different?

Because participants complete unmoderated studies on their own time and an online tool automatically captures, then presents the data to the researcher, it’s easy to include hundreds or even thousands of participants without investing the time and effort moderated sessions require. The main challenges are finding a very large number of potential participants, ensuring that they are representative, and convincing them to participate. This requires very different recruiting methods than those we typically use for traditional, moderated user research.

Three Methods of Recruiting Participants

There are three ways you can invite people to participate in an unmoderated study. You can

  • email a link to your study to potential participants
  • use a recruiting company’s online panel of participants
  • use online intercepts

The appropriate method to choose depends on the specifics of your study and the type of people you are trying to recruit.

Emailing Potential Participants

Emailing a link to your study is an effective method if you can easily identify potential participants and obtain their email addresses. However, emailing a large number of people can be time consuming.

Gathering a List of Potential Participants

Existing lists of email addresses—such as customer lists, membership lists, or employee directories—are good places to gather a list of potential participants. Ideally, these lists offer enough information so you can narrow them down to the type of participants you want. Even after narrowing the list, you may need to do some additional screening.

Some unmoderated, remote user research tools can present screening questions at the beginning of a study and reject respondents who don’t fit your profile. The risk here is that each question adds to the length of the study, increasing the chance that people will abandon it. Be careful to include only screening questions that are absolutely necessary.

Another way to build a list of potential participants is to use an intercept questionnaire that lets you recruit visitors from your Web site. With screening tools like Ethnio, you can set up an intercept on your site to invite visitors to participate in future studies. Those who want to volunteer must take a screening questionnaire, then the tool adds those who qualify to a database of participants you can use when emailing invitations for future studies.

Determining How Many People to Invite to Participate

The ideal number of participants depends on your research goals. You should determine the minimum number of participants you need for acceptable results and, if necessary, a maximum number. The minimum helps you know when you’ve received enough responses to end the study. The maximum is also helpful, because some tools allow you to automatically end a study when you reach a set limit. This can prevent you from paying too much in incentives.

Because mass emailing of invitations to studies often results in low completion rates, you should send email messages to far more people than a study requires. Some of the messages will get blocked by spam filters, only some recipients will open and read them, only some of those who read them will click the link to the study, and fewer still will actually complete the study. Response rates are often as low as five percent. So it’s best to assume a low response rate and, therefore, invite an appropriately large number of potential participants. For example, if you want 100 participants to complete a study, invite 2,000 people—assuming a five-percent response rate.

Creating an Effective Email Message

Craft a short, effective email message that describes your study, highlights the incentive, and provides a link to the study. To keep your message brief, don’t include detailed instructions about how to complete the study. Instead, save those details for the instructions at the beginning of your online study.

Establishing Credibility

Your email message must quickly establish credibility and trust with potential participants. Understandably, people are wary of clicking links in messages from people they don’t know. Although mass emailing requires less work, sending messages to individuals is more effective in establishing trust and, therefore, it is less likely people will ignore your messages.

Personalize your email messages by

  • sending each individual person a message, instead of sending one message to many email addresses
  • using the recipient’s name at the beginning of your message
  • writing in a friendly, professional tone
  • describing who you are and who you’re representing
  • including your name, title, company name, company Web site link, and contact information at the bottom of your message

Who an email message comes from is very important in establishing credibility. Because a message from a familiar source gets more responses, it’s a good idea to have a client for whom you’re doing user research send the email messages. For example, employees are more likely to trust an internal email message from a fellow employee, and customers are more likely to trust a request from a company they patronize than a request from an unknown third party.

If you must send the email messages yourself, mention the name of someone with whom the recipient is familiar at the beginning of the message. If you are recruiting employees, providing your client’s name and contact information is the next best thing to having your client send the messages directly. Although few recipients will actually contact you or your client, giving them the ability to do so increases their trust in you and the likelihood they will respond positively.

Providing an Incentive

Participants are willing to accept lower incentives for unmoderated, remote studies, because they require much less time and effort than moderated studies. Still, with so many participants, it can be expensive to offer individual compensation to every participant. Plus, because the researcher and participants never meet, logistical difficulties arise in getting compensation to all of the participants.

Therefore, an economical alternative to making a small payment to each participant is offering a drawing for a prize. The chance to win something desirable like an iPad or an iPhone is much more enticing to many people than receiving a five-dollar check. So, for a few hundred dollars, you can get hundreds of participants interested in your study. However, there are legal requirements for drawings, so be sure to check with a lawyer first.

Remember to offer an incentive that is appealing to all potential participants to avoid skewing the type of participants you get. For example, a drawing to win a video-game system may appeal to certain people, but not others.

Monitoring Participation and Sending Reminders

After sending your email messages, monitor the number and types of people who have completed your study. If, after two days, you haven’t reached your goal for participation, send a reminder on the third day. This usually results in your getting additional participants. If you still need more participants, send an email message to a new set of people you haven’t yet contacted.

Using an Online Panel

If you think you’ll have difficulty finding participants on your own, an online panel of participants is a good option. A panel is a database of people who have volunteered to participate in online surveys and other market research studies. Many traditional recruiting companies provide online panels in addition to recruiting individuals for in-person studies.

Screening the Participants

As with in-person studies, you must tell the recruiting company what type and number of participants you are seeking. However, because the recruiting company has already prescreened its panel members, a traditional screening questionnaire is not necessary. Recruiters can filter their database according to the characteristics you need.

If your requirements are more specific than the information that is available in the recruiting company’s database, you can add a screening questionnaire at the beginning of your study. Some unmoderated user research tools let you create screening questionnaires. Alternatively, you can have the company providing the online panel create a screening survey for you.

Ensuring the Quality of Participants

An advantage to using an online panel is that you’ll get people who are highly motivated to participate in online studies. Because they are being paid and want to be invited to participate in future studies, they are motivated to take the study seriously and provide thorough feedback. However, you should consider that the type of person who volunteers to participate in online studies may be different from your typical users.

Providing Incentives and Fees

Using a panel is more expensive than doing the recruiting yourself, but it saves you a lot of time. The recruiting company collects a recruiting fee and incentive from you for each person who completes the study. For example, if they invite 1,000 people, but only 100 complete the study, they will charge you only for those 100 people. Although the recruiting fee and incentive is much lower per participant for panels than for in-person studies, these costs add up with a large number of participants.

Some unmoderated research tools allow you to specify a maximum number of participants for a study. This is an important feature to use with a panel, because it will limit your costs to only the desired number of participants.

Using Online Intercepts

Online intercepts pop up on a Web site and invite people to participate in a study. After a brief screening questionnaire, they take qualified participants directly into the study. Most of us are familiar with survey invitation intercepts, but you can use intercepts for any type of online study. Intercepts are popular with user researchers, because recruiting occurs automatically and can continue indefinitely.

Creating an Intercept

Some unmoderated user research tools provide the ability to create intercepts. If yours doesn’t, you may need a developer to create the intercepts and tie them into a Web site.

Ensuring the Quality of Participants

Many people like using intercepts because they recruit real users as they are in the process of actually using a Web site. Unlike a panel, they aren’t people who have volunteered and are eager to participate in research. So they may be more representative of the average user. On the other hand, few Web site visitors take the time to participate in research, so those that do may not be representative of its average user either. 

Intercepts are much less personal than receiving an email invitation or volunteering to be part of a panel, so participants may be less motivated to complete a study. Plus, they may be more likely to give up if the study takes too long or requires too much effort.

Doing Ongoing Research

Because intercepts can remain on a Web site indefinitely, continually recruiting participants, they are ideal for long-term, ongoing research. Once you’ve set them up, you can periodically check the results. Other than that, very little additional effort is involved.

Providing Incentives

Because it’s difficult, both logistically and financially, to provide individual incentives to intercept participants, it’s common to offer drawings for a prize, which have proven to be fairly effective incentives. Again, check with a lawyer about the legal considerations involved in holding a drawing.

Choosing Among These Three Methods

Here are some general guidelines for choosing among these three methods of recruiting participants for unmoderated studies:

Email participants if…

  • You can easily obtain a list of potential participants and their email addresses, either from existing lists or by recruiting participants on your Web site.
  • You don’t need to do much extra screening of potential participants.
  • You can have someone participants know send the email messages, or you can use a trusted party’s name to increase credibility and trust.
  • You have more time than money.

Use an online panel if…

  • You can’t easily get a list of potential participants and their email addresses.
  • You need a very specific type of participant, which would require a lot of screening.
  • Your study requires participants to spend a lot of time and effort to complete it.
  • You have more money than time.

Use online intercepts if…

  • You can’t easily get a list of potential participants and their email addresses.
  • You want to do research with your site’s current users.
  • You intend to run ongoing studies on your site.
  • You have the time to wait for enough responses to accumulate.

In Conclusion

Unmoderated, remote user research provides the ability to reach a large number of participants, but finding those participants and convincing them to participate is challenging. By following the tips this article has provided, you’ll get more and better participation for your unmoderated, remote studies. 

Principal UX Researcher at AnswerLab

Philadelphia, Pennsylvania, USA

Jim RossJim has spent most of the 21st Century researching and designing intuitive and satisfying user experiences. As a UX consultant, he has worked on Web sites, mobile apps, intranets, Web applications, software, and business applications for financial, pharmaceutical, medical, entertainment, retail, technology, and government clients. He has a Masters of Science degree in Human-Computer Interaction from DePaul University.  Read More

Other Columns by Jim Ross

Other Articles on Remote UX Research

New on UXmatters