Moderated Versus Unmoderated Usability Testing

March 7, 2022

Usability testing is the practice of evaluating a product such as a Web site, application, system, or platform as real people are performing tasks with it. We put these products into the hands of real users to see whether they can navigate the product easily and complete the necessary tasks. At various phases of the software-development process, we can conduct usability testing on prototypes, alpha and beta versions of a product, and even live Web sites.

To get the most out of your usability testing, you must first determine which method of testing is most appropriate for your project. There are two broad types of usability-testing methods: moderated and unmoderated usability testing. In this article, I’ll discuss the benefits and drawbacks of both moderated and unmoderated usability testing and provide some best practices for each testing method that can ensure successful test sessions.

Champion Advertisement
Continue Reading…

Moderated Usability Testing

This usability-testing technique entails the active engagement of a trained facilitator or moderator, as well as the use of a test script that directs participants on what tasks to perform during their test session. The moderator guides test participants through a series of tasks. The moderator and participants can communicate directly during these sessions, and the moderator controls the sessions. This approach is ideal for testing complicated prototypes that are likely to elicit a lot of questions from participants, as well as design solutions that have security requirements.

Pros of Moderated Usability Testing

Let’s first consider some pros of moderated usability testing, as follows:

  • This type of testing leads to a higher level of participant engagement. The moderator can engage in casual conversation with test participants, which helps to build trust and motivates participants to complete the tasks.
  • The direction of the test session is mostly in the moderator’s hands. The moderator can pose questions directly to the participants and have the participants respond.
  • This method enables UX researchers to ask follow-up questions and obtain further information from the participants about the usability issues they’ve encountered. The moderator can delve more deeply and inquire about the reasons why they were having difficulties. For example, if a participant gets stuck, the moderator might ask what they expected to happen or what the problem was.
  • Moderated usability testing is also a good approach for detecting subtle responses and behaviors of the participants such as body language. If a participant appears confused, the moderator can figure out what problem the participant is experiencing. Because UX researchers can obtain input from participants in real time, this helps them prepare a more detailed report.

Cons of Moderated Usability Testing

Now let’s look at some cons of moderated usability testing, as follows:

  • Because others are watching them, participants may act differently from the way they would act when they are alone. This is called the observer effect. For example, an observer might have a specific response or make a clever retort to the moderator.
  • For best results, moderated usability testing requires more resources and time. It requires that you hire test participants who have the same characteristics as the target users, use skilled moderators, and provide a private location for the testing.
  • A moderator could ask a question in a way that biases the discussion.

Best Practices for Moderated Usability Testing

To ensure the success of your moderated usability-testing sessions, employ the following best practices:

  • Define goals and the scope of testing. You’ll need to figure out which components of your product you want to test, as well as your main testing goals and the tasks participants should complete to meet each goal.
  • Clearly define the selection criteria for test participants. To acquire the most meaningful insights, make sure you get test participants belonging to the correct demographic. If you’re wondering where to find these people, check out this article on recruiting usability-test participants for free.
  • Conduct moderated usability testing according to the right process. Sessions should follow this sequence:
    • pre-session questions—Inquiring about participants’ past and general behavior will aid in the identification of common habits and other personal facts about your target audience.
    • actual tasks—The test participant performs a list of tasks using the product. All tasks should be selected according to the goals of the testing.
    • post-session questions—Ask participants general questions about their experience and gather feedback on the product. Post-session questions give test participants the opportunity to share their thoughts and ask questions regarding the product.
  • During the testing, urge test participants to think aloud. It’s a good idea to have them talk through what they’re doing as they’re doing it. When test participants think aloud while completing tasks, the moderator can better understand the reasoning behind each decision, revealing the participant’s motivations and opinions.
  • Take notes during the test sessions. You can learn a lot from your participants, and you won’t be able to remember everything that happens. Therefore, it’s a good idea to jot down any intriguing observations or data you come across so you can refer to your notes later while analyzing the test results.
  • Analyze the findings and present them in a report. Write a report that contains the most significant findings and assists the product team and stakeholders in prioritizing design decisions. Use simple, easy-to-understand language and images to explain the facts in your report.

Tips for the Moderator

Following these tips can help the moderator achieve better outcomes:

  • With the approval of your participants, record the sessions.
  • Allow participants to deliberate and complete their tasks without any undue influence such as leading questions or suggestions.
  • Don’t draw any conclusions during the sessions. Wait until you’ve gone over all the data from all the sessions.
  • Keep an eye out for participants’ gestures and expressions, which provide much of the input you need.

Unmoderated Usability Testing

This is a usability-testing technique in which the test sessions are not monitored and no one is present when the participants answer questions or complete tasks. Unmoderated testing allows participants to complete the tasks or provide their answers at their own pace—and at a time and location of their choosing. Unlike moderated usability testing, participants cannot receive real-time assistance. Because of the lack of interaction with a moderator and the inability of participants to raise inquiries, this testing approach is recommended for less complex prototypes.

Pros of Unmoderated Usability Testing

Let’s consider some pros of unmoderated usability testing, as follows:

  • These test sessions can be completed at any time and anywhere—whether the participant is at home or on a mobile device—making it a very convenient usability-testing method.
  • This test method provides access to a larger geographical sample and, therefore, enables you to test a product with a wider audience.
  • Unmoderated testing is less expensive and time consuming to conduct and requires fewer resources.
  • There is less risk of bias, and the observer effect—individuals’ acting differently when they are aware that they are being watched—does not apply.

Cons of Unmoderated Usability Testing

Now let’s look at some cons of unmoderated usability testing, as follows:

  • During the sessions, no researcher is present to guide the participants. Therefore, if participants get stuck, run into technological difficulties, or don’t understand the questions, no one can help them. This could have negative impacts on the test results.
  • Another disadvantage of the lack of real-time interactions with participants is the risk of their skipping through the early tasks and moving straight to the follow-up tasks. Participants may diverge from the testing flow that you had planned because they are receiving no guidance.
  • Data collection is limited. Only questions that a UX researcher has prepared in advance get answered. There is no opportunity for a researcher to ask any follow-up questions during a session. Therefore, before the start of the sessions, you must thoroughly prepare all of your questions.
  • You’ll miss out on observing the participants’ subtle responses to certain test questions—such as hesitancy when answering specific questions.

Best Practices for Unmoderated Usability Testing

To ensure the success of your unmoderated usability-testing sessions, employ the following best practices:

  • Because there won’t be any moderator to guide the participants, make sure the test activities are straightforward and easy to understand, with clear instructions on how to approach them. This eliminates the possibility of a participant’s misunderstanding a task and, therefore, performing it incorrectly, which could distort the test findings.
  • Increase the number of test participants to avoid the risk that some of them might not show up, which sometimes happens in unmoderated usability testing.
  • Thoroughly analyze the data and present them in a report that includes your most significant insights. Use simple, easy-to-understand language and images to explain the facts in your report. Your report can assist the product team and stakeholders in prioritizing design decisions.

Moderated Versus Unmoderated Usability Testing

When to conduct moderated usability testing:

  • Use this approach when you have the necessary time and resources.
  • You need in-depth feedback from your users.
  • During the earlier stages of design, when your team is working on a product concept, this method can help you explore various solutions.

When to conduct unmoderated usability testing:

  • Use this approach when tasks are straightforward and participants would be unlikely to require any guidance during their test session.
  • You’re testing an already functional Web site or high-fidelity prototype.
  • You’re on a tight deadline, because you could conduct unmoderated tests with hundreds of participants, within a short period of time.

Which Testing Approach Should You Use?

There are both advantages and disadvantages to both moderated and unmoderated usability testing. Moderated testing enables UX researchers to communicate directly with their test participants, allowing them to gain a better understanding of users’ problems and, hopefully, address them.

Unmoderated usability testing is a rapid, effective way of gathering data from your participants. Because of the quick turnaround, firms can attract a higher number of participants, providing a larger test sample.

In specific situations, you can conduct either or both moderated and unmoderated usability testing. You could begin with an unmoderated session, during which the participant answers questions whose answers would enable you to address the most important usability concerns. Then you could switch to a moderated session, asking questions to home in on specific issues the participant raised during the first session and gain more insights into them. 

Content Creator at UXtweak

Dar Es Salaam, Tanzania

Salome KatunziSalome is still in the early stages of her product-design career, but she has already worked with a variety of multinational companies to design user-friendly solutions. She empathizes with users to identify their problems and come up with innovative solutions. Salome also enjoys writing educational content about user experience and visual design. She does most of her writing for UXtweak, a renowned UX research platform.  Read More

Other Articles on Usability Testing

New on UXmatters