In my previous Continuous Research columns about user-research panels—user panels, in short—I covered the topics of automating your user-recruitment processes and ways to fill your recruitment pipeline. I argued that we should consider recruitment a high-value first step in the user-research process rather than the task checkbox that UX teams too frequently perceive it to be. If you automate your processes and focus your attention on building lasting personal relationships, you can have a group of trusted advisors who feed you insights continuously. In this final column in my three-part series about user panels, I’ll discuss how you should engage with new panel members.
All your efforts have worked out and you now have a steady stream of HiPPO-quality user-research participants who want to give you feedback. This is a great problem to have! Now you can focus on building your relationships with these people. The way you welcome them, what information you store about them, and the way in which and how regularly you interact with them determine how successful you’ll be at building this group of trusted advisors.
Empathy is absolutely key in all these interactions. Do your new panel members understand the benefits of continued engagement with your organization? Are they interested specifically in a single feature? Are they happy to spend time with you at regular touchpoints? What are the priorities for them in these interactions? Get answers to these questions and work with your panel members according to what you learn, and you can make these interactions as enjoyable as you want them to be useful.
What You Can Do
Next, I’ll cover a handful of practical tips that I’ve picked up along the way—both by talking with researchers and product-team members and through the research that I’ve conducted. Some of these tips came from prospects and clients of Airtime, while others are the direct result of our product-development effort. What is common across this motley set of ideas is their instant actionability, along with the fact that they make the experience of your user-research panel members pleasant, while streamlining their journey.
Welcoming New Members
Having people sign up for your user panel doesn’t mean that they’ll actually participate. Wear your sales hat for your initial interaction with them. Communicate the benefits of participation to your new panel members, and make them feel like a member of a special club. Help them understand how they can engage with you right away and in the future. Use short messages in communicating with them. If you have a lot to say, do it in a series of welcome email messages or in multiple email messages, over a few days or weeks. These people are giving you their time and attention for free, so it’s best not to abuse their trust.
Waiting to Do Profile Surveys, Eligibility Criteria, and Tagging
Depending on how you’ve recruited your new user-panel members, you might already know a lot about them or you might know very little. If you’re recruiting an existing client to participate, you can easily mine data on that client from your Customer Relationship Management (CRM) system and software-usage data. An additional benefit of this data is that it’s less biased than survey data or answers to user-interview questions and you know that it’s completely factual.
If you’re recruiting prospects and know very little about them, obtain only the bare minimum of information about them before their first interview. Save additional requests for information for your first few interactions. This helps keep the process easy for them. If you streamline the process, you can avoid the attrition among your new panel members that could occur if you ask too much of them. For example, you should get basic data pertaining to their eligibility, but hold your profiling questions for later. Profiling is useful when tagging and segmenting your panel, but it’s not a must-have before your first chat.
Getting New Members to Engage Right Away
Many companies let users join user panels just for the sake of it, then invite them to participate in their next research project, which might be months away. If you engage with new members too much later, you’ll lose precious momentum and your new members might lose interest. When you have a continuous-research practice, with frequent light touchpoints with your users, you’ll always be able to recruit participants for specific interviews.
Recruiting participants for interviews sustains the momentum. Plus, it gives you the opportunity for a personal welcome, lets you find out anything you might have asked in a profile survey, and minimizes no-shows. The earlier the interview appointment, the better.
No-shows are a real problem, and you can avoid them by booking your interviews early, sending several reminders, and choosing the right channels for your interviews. One study at Airtime faced many no-shows by senior people because they weren’t tech savvy and were uncomfortable using Zoom, so they switched to phone calls. For another research project, having screen sharing was crucial, so the researchers dedicated an extra 15 minutes to each session to call respondents, then set up the Zoom connection together on the phone.
Preparing for the First Session
The first interview session is an excellent opportunity to set the tone of your new relationship, find out a lot of personal things about your new panel member, and of course, get the feedback you need for your ongoing research. If you’re running a collaborative research practice—and you should be to gain the numerous benefits I outlined in my blog post “How Collaborative User Research Breaks Down Organizational Silos”—you’ll include other stakeholders from the product team as observers.
Before the session, define the key learnings you want to get out of the session, then share them with your stakeholders so they can provide feedback on them or suggest their own. Prioritize them as necessary.
It is an excellent practice to ask participants to write down their answers to these key questions in advance. Comparing their assumed answers to their actual answers after the interviews can uncover important assumptions and biases within the product organization.
Distribute tasks. Simply sitting in a meeting is much less effective than actively participating. For example, the user researcher could lead and facilitate the sessions, the product manager could take notes, and the designer could note down metacommunications from the participant. Alternatively, the observers could all write down their own key learnings from what the participant says. It would be interesting to compare whether these key learnings match and, if not, why not. Then everyone can ask questions at the end of the session, during an open Q&A.
Professional panel providers recommend engaging with your user-panel members at least once a month. On the other hand, theirs are panels for which respondents get paid for participating. In my previous column, I explained why I avoid paid-panel providers. They usually make recruitment a transactional, one-off event; their filters are not sufficiently granular; and they cost a lot. Of course, these are companies with deep, user-research cultures, who have already built a panel of trusted advisors. One such company, a leader in software development for the global music industry, rotates quarterly Net Promoter Score (NPS) respondents to avoid burnout. As a result, they ask their NPS respondents to participate just twice a year at a maximum.
There is no silver-bullet answer to the ideal contact frequency, so let’s take a more generic approach. A top-down point of view looks like this:
Ideal contact frequency for you to contact each panel member = [frequency of your releases that need testing] * [number of people on your panel] / [minimum number of people you must ask for each release] * [willingness to respond as a %]
For example, if you do monthly releases and want to ask ten people to participate in your research for each release, you have 100 people on your user panel, and you need to ask 20 people to get ten to participate, the equation looks like this:
[monthly releases] * [10 you want to ask] / [100 full panel population] * [50% response willingness] = monthly * 10 / 100 * 50% = you contact each panel member every five months
A bottom-up approach to getting a sense of panelists’ willingness to participate is to communicate directly with each panel member. Or you could assess activity levels on channels such as your product forum. You can grade each person’s willingness on a scale of three or five and contact the more willing participants more frequently.
Ideally, the top-down approach—that is, recruiting on the basis of what you need—and the bottom-up approach—which depends more on what panelists need—should meet in the middle, so you can contact each panel member as often as is convenient for them, while getting enough feedback for each of your releases.
Channels That Build Community
In addition to direct, one-on-one contact channels such as video interviews and surveys, consider complementary community channels as well. Community forums or chat channels such as Slack encourage interactions between your panel members. This could enable you to uncover insights that are unknown unknowns to you.
Chats are great for more synchronous discussions and nearly real-time interactions, while forums provide an asynchronous experience. These discussions tend to be more structured because of their clearly delineated topics, up- and down-voting capability, and better search. Tools such as Zenloop use Natural Language Processing (NLP) to analyze and classify forum insights. NLP algorithms are good enough to indicate sentiment or the key themes of a discussion. You could subsequently organize sessions with users for deep dives into key themes. You would need about 100–150 active participants in a forum for participants to generate meaningful, but not overwhelming amounts of content.
Don’t forget to mention these communities during your first interactions with new panel members, so they can immerse themselves right away.
If you want to make the most of your user panel, you’ll need to treat each member as a valued friend. Personalized, albeit resource-heavy relationships form the basis on which you can build your product advisory panel. If you can set up an automated recruitment pipeline and focus your efforts on building these key personal relationships, you’ll be able to understand your market on the go, in perpetuity. This is the equipment you need to build the best product in whatever sector you work. Good luck!
At Airtime, a software company that develops collaborative user-research tools and helps organizations implement collaborative-research practices, Peter heads up operations. Peter and his two founding partners launched their user-research platform because they felt that many products—whether digital or physical, business to consumer (B2C) or business to business (B2B)—are less than optimal and could be improved through the simple exercise of listening to customers. Peter has a diverse background—from investment banking to institutional sales to working for asset managers, asset owners, and broker-dealers in Europe, Asia, and the Middle East. For seven years, Peter worked for the leading international provider of equity indices and investment analytics. Following his stint in the financial industry, Peter switched to freelance startup consulting and investment-analytics publishing. Peter has a Masters in Capital Markets from the Corvinus University of Budapest. Read More