Fortunately, there are ways to improve your recruiting success. In this column, I’ll share tips for writing a better screener, eliminating professional user research participants, minimizing no-shows, deciding who should do the recruiting, and what to do when the wrong people slip through your screening process. These tips are valuable to those who are new to user research, as well as to experienced researchers as a refresher on a part of the user research process that people often take for granted.
Write a Better Screener
Because user research is only as good as the participants you involve, the screener is one of the most important documents for a user research project. Yet it’s often the one people tend to take for granted. Few books or schools detail how to write an effective screener, and as a result, we often learn to write screeners through trial and error, rely on screeners the marketing department has created, or reuse screeners from previous projects without considering whether they are still appropriate. The following tips will help you create a better screener.
Prevent People from Hanging up on You
Unless you have good connections to potential participants—for example, through your client or a participant database—either you or your recruiter will be making cold calls to them. When you make cold calls, people will assume you’re a telemarketer, so you need to get right to the point and quickly establish that you’re not selling anything. Say you’re recruiting people for a paid study—mentioning the money up front gets people’s attention and keeps them on the phone.
Don’t Use the Screener to Gather Information
The purpose of a screener is to select the best participants for a study. The more questions you ask, the longer and more cumbersome a screening call becomes. So don’t include information-gathering questions unless they also serve a screening purpose. Asking participants to fill out a questionnaire at the beginning of a research session is a much better way to gather information.
Ask the Elimination Questions First
Don’t waste the time of either the recruiter or potential participants by making them go through a lengthy screener before getting to the questions that eliminate the most people. Ask those questions first, so only the most likely candidates must go through all of the questions.
Eliminate Conflicts of Interest
Eliminate potential participants who may have a conflict of interest with your client or who have too much insider knowledge. For example, if you were conducting usability testing on an airline’s Web site with travelers, you should screen out people who work for that airline or a competitor.
Recruit Based on Behavior and Attitudes
When recruiting for small-sample user research studies, the importance of behavior and attitudes surpasses that of demographics. For example, when doing user research on a flooring company’s Web site, it is far more useful to recruit homeowners who are in the market for flooring than it is to simply replicate the company’s customer demographics.
Consider what attitudes should qualify or disqualify someone from participating in your study. For example, a person who has strong loyalty to mom-and-pop shops and a dislike of big-box stores would not be a good participant for usability testing on Walmart’s Web site. In such a case, you should ask potential participants about the stores they frequent and their attitudes toward certain retailers.
Screen for Computer and Web Experience
Ensure that the participants’ experience with the computer and the Web matches that of your user groups. You’ll often want to eliminate people with too little or too much computer and Web experience, unless their level of experience is appropriate for your project. For example, in a usability test, you don’t want participants to confuse user interface problems with the problems new computer users face in general. Similarly, you won’t usually want to test with participants who are Web developers or UX designers. They bring a level of expertise and a focus that is not representative of a typical user.
When you are screening, don’t ask participants to assess their own computer and Web experience. Self-assessments are very subjective, and some people are reluctant to admit their inexperience. Instead, ask specific questions like whether they have a computer at home, their Internet access, how many hours they spend on the Internet per week, how many years they have been using a computer, and the type of computer activities they perform regularly. Then make your own assessment of their experience.
Eliminate the Strong, Silent Types
In a qualitative study, there are few things worse than a participant who gives only one-word answers. It requires a lot of work to drag useful information out of such people. To determine how expressive people are, ask a few open-ended questions that relate to the topic of your study.
Ensure That People are Physically Able to Participate
It’s obvious—but easily overlooked—that participants must be physically able to participate in a study. For example, in most cases, unless you’re doing usability testing for people with disabilities, participants need to be able to read a computer screen. In this case, you should ask, “Can you read a computer screen, using contact lenses or eyeglasses if necessary, without difficulty?” And remind scheduled participants to bring their contacts or glasses.
Eliminate the Usual Suspects
Recruiting companies sometimes rely on their participant databases before calling fresh participants. While many of these people are great participants who are very good at providing insightful feedback, others volunteer a little too often. They may be supplementing their income as a professional user research participant. (I’ll discuss this in greater depth in the next section.) To avoid the usual suspects, eliminate those who have recently participated in a study—for example, within the last six months.
Screen Out Professional User Research Participants
People who frequently supplement their income by participating in user research will say and do whatever it takes to get into a study. It’s often all too easy to figure out the correct responses and avoid being eliminated. However, there are ways to improve screener questions and eliminate those who stretch the truth to get included in a study.
Ask Open-Ended Questions
With multiple-choice questions, it’s often easy to guess which answers to choose. For example, consider this question:
“When was the last time you participated in a focus group or usability study?”
|[ ]||6 months ago or less|
|[ ]||More than 6 months ago|
It’s obvious the recruiter is looking for people who have not participated in a study for the last six months. Instead, ask the open-ended question, without reading the choices to potential participants, making it more difficult for them to determine the elimination criteria.
Include Multiple Elimination Answers for Questions
When it is necessary to ask multiple-choice questions, it’s possible to make the elimination answers less obvious by including multiple elimination answers for a single question. For example, if you’re trying to recruit people who watch 13 or more hours of TV per week, the safe answers to the following question seem obvious.
Approximately how many hours of TV do you watch per week?
|[ ]||12 or less||TERMINATE (eliminate the person from the study)|
|[ ]||31 or more||CONTINUE|
Because the middle three answers are so specific, they seem like safe choices to avoid being eliminated. It’s obvious that 12 or less or, possibly, 31 or more are the cutoff points. The wording of 12 or less and 31 or more implies that, if you watch that little or that much TV, the recruiter doesn’t care about the exact number of hours you watch, and you’ll be eliminated.
Instead, provide more than one elimination answer to hide the obvious cutoffs. For example, in the following question, it’s much less obvious which answers will result in elimination.
Approximately how many hours of TV do you watch per week?
|[ ]||0–3||TERMINATE (eliminate the person from the study)|
|[ ]||4–6||TERMINATE (eliminate the person from the study)|
|[ ]||7–9||TERMINATE (eliminate the person from the study)|
|[ ]||10–12||TERMINATE (eliminate the person from the study)|
|[ ]||40 or more||CONTINUE|