Remote Usability Testing: Thinking Outside the Lab

By Rebecca Baker

Published: April 21, 2014

“In the world of user experience, usability testing may be one of the hardest things to sell, while simultaneously being one of the most critical elements of ensuring an application’s usability.”

In the world of user experience, usability testing may be one of the hardest things to sell, while simultaneously being one of the most critical elements of ensuring an application’s usability. Because usability testing is relatively expensive and time consuming, executives and development leads who have never observed a test session often dismiss testing as unnecessary and devalue the data that comes out of usability studies. In many sectors, the perception is that usability testing requires a dedicated lab with one-way glass, video cameras, and expensive microphones—all to observe users who a company has flown in at great expense and trouble to perform a couple of tasks and complain about their user experience.

However, with advances in technology over the last ten years, the ability to do usability testing with remote users has increased significantly, both reducing the investment in time and money and expanding the reach of researchers. Remote usability testing lets us get access to users from across the globe, ensuring a more representative set of results. This is vital in our hyper-connected reality. And we can now accomplish this without traveling or having a dedicated lab. Remote usability testing is a welcome addition to any researcher’s arsenal—particularly as we try to fit data collection into agile sprints and work within the constraints of an increasingly lean bottom line.

Unmoderated Remote Usability Testing

“Unmoderated remote testing focuses on obtaining data about the usability of individual workflows and experiences rather than capturing broad, overarching metrics about general usage.”

Unmoderated remote testing is one of the most popular methods of testing Web applications. Many confuse such testing with the use of Web analytics tools such as Google Analytics and Clicky, but unmoderated remote testing focuses on obtaining data about the usability of individual workflows and experiences rather than capturing broad, overarching metrics about general usage.

In unmoderated remote tests, participants complete predetermined tasks using an application. Services like Loop11 or UserTesting let us collect data about how long it took participants to complete a task, where they made mistakes or referred to the Help system, whether they were able to complete the task, and how they navigated the application to complete the task. Researchers can analyze the results and provide specific feedback on the product design.

Pros

“Unmoderated remote testing lets you recruit and test using large numbers of participants in a relatively short timeframe.”

Unmoderated remote testing lets you recruit and test using large numbers of participants in a relatively short timeframe. Because remote testing does not rely on a moderator’s conducting the test sessions, participants can work through your test tasks simultaneously. Large amounts of data can be very powerful when you’re making a case for a significant and potentially costly change to existing software. If you’re working in a Lean UX environment, unmoderated testing gives usability a foot in the door, providing immediate value at a low cost. An unmoderated remote–testing service generally handles participant recruitment—although you can take this on yourself—and scheduling is unnecessary. In addition, you can easily test competitors’ products, creating an opportunity to present side-by-side results. You can easily fit unmoderated testing into an agile sprint schedule and provide timely feedback for the next sprint.

Cons

“With no means of obtaining participants’ feedback aside from comment fields or a questionnaire that you administer at the end of a test session, it can be difficult to accurately interpret the reason why a participant has made a mistake or misstep.”

Unmoderated remote tests require a fully functional product or prototype, so you cannot use this approach when testing concepts through a cognitive walkthrough. Unmoderated testing often leaves the researcher to wonder why? With no means of obtaining participants’ feedback aside from comment fields or a questionnaire that you administer at the end of a test session, it can be difficult to accurately interpret the reason why a participant has made a mistake or misstep.

For example, an unmoderated test might show that several participants were unable to complete a shopping cart workflow, abandoning the task at the last step before committing the purchase. Was it difficult to see how to complete the purchase? Did participants think they had submitted their order, so were finished with the task? Did a cat sit on a participant’s keyboard and cancel the transaction? Did the process take too long, so participants got bored and quit? In unmoderated tests, these questions remain unanswered. A participant’s voice is silent for the developer and executive as well. Hearing and seeing someone’s frustration and confusion during a test is invaluable. Non-researchers’ hearing such visceral feedback can make the difference between fixing a problem and merely accepting it as it is.

From an enterprise perspective, unmoderated remote testing may be inaccessible to large companies because of legal considerations, the scale and complexity of their software, or the need to test with a limited and highly specialized user base. Many large B2B (Business-to-Business) companies have significant legal exposure when testing their software with users, requiring lengthy non-disclosure agreements and assurances of confidentiality. Remote testing services work on the assumption that the applications to be tested are Web based and consumer oriented. Neither of these assumptions is valid for most large B2B companies—many of which still maintain thick-client, complex software that is geared toward an elite user base such as security administrators or systems administrators.

Is Unmoderated Remote Testing for You?

“Unmoderated remote testing is generally best for assessing targeted areas of a user interface that have well-known issues, A-B testing, simple competitive analyses, collecting data in large numbers, and initial problem discovery….”

Unmoderated remote testing is generally best for assessing targeted areas of a user interface that have well-known issues, A-B testing, simple competitive analyses, collecting data in large numbers, and initial problem discovery—assuming you follow up with moderated testing to uncover the why. Unmoderated testing is not recommended for solving complex or poorly understood issues or as a substitute for discovery work such as user interviews or cognitive walkthroughs.

Moderated Remote Usability Testing

“Moderated remote usability testing lets participants complete predetermined tasks using an application, while interacting with a moderator who prompts them for feedback during the test, using a think-aloud protocol.”

A less well-known, but very useful testing technique, moderated remote usability testing lets participants complete predetermined tasks using an application, while interacting with a moderator who prompts them for feedback during the test, using a think-aloud protocol. You can collect data about how long it took participants to complete a task, where they made mistakes or referred to the Help, whether they were able to complete the task successfully, and how they navigated the software. You can also discover why issues arose, whether different participants encountered the same problems when using the software, what other problems participants experienced, and what features they desire that would improve the product. Numerous tools are available for moderated remote testing, but setup and hosting of the software that you are testing is generally on your own network.

Pros

“Moderated remote testing lets you recruit and test using participants from all over the world, without the expense of travel.”

Moderated remote testing lets you recruit and test using participants from all over the world, without the expense of travel. Difficult to schedule users become easier to reach because they can complete the test session from their own office, eliminating the travel time to and from your lab. Because a moderator conducts the tests, you can collect significant amounts of qualitative and contextual data—often more than you’ve originally anticipated. And you can invite as many observers as your bandwidth permits, without making your participants feel uncomfortable or nervous. You can often fit moderated testing into an agile schedule—but not for every sprint.

Cons

“The think-aloud protocol can be difficult for some participants who are perhaps more reticent by nature.”

Moderated remote testing can present significant setup challenges, especially in regard to performance. For example, if you are using a tool like WebEx to let participants control an application on your desktop, there can be a delay between their selecting an option and seeing a result. This can lead to false errors that are not representative of how a user would actually interact with the software.

The think-aloud protocol can be difficult for some participants who are perhaps more reticent by nature. The moderator must be skilled in remote testing to ensure good results. The inability to see facial expressions and body language, as well as the potential for interruptions, can make remote testing less ideal than in-person testing in a lab, where you can more closely control the environment. For example, participants could be texting during their test session, their cube mate might be having a loud discussion with coworkers, or a fire alarm could even go off.

From an enterprise perspective, moderated remote testing may also present legal issues. Because participants could potentially take screenshots, most legal departments require lengthy non-disclosure agreements and assurances of confidentiality.

Is Moderated Remote Testing for You?

“Moderated remote testing is generally best for studying complex or poorly understood issues, when it is important to understand why a participant is doing something.”

Moderated remote testing is generally best for studying complex or poorly understood issues, when it is important to understand why a participant is doing something. It is also good for interviews when you’re gathering requirements and cognitive walkthroughs when you’re exploring an idea or concept with potential users before creating a fully functional prototype.

What About Mobile?

“While remote usability testing on mobile devices is still in its infancy, there are some promising testing tools currently on the market.”

While remote usability testing on mobile devices is still in its infancy, there are some promising testing tools currently on the market. Unfortunately, most of them are still in the realm of unmoderated testing or require you to preload an application on a mobile device for participants to use, making these tools inappropriate for remote testing. How to provide your application on a mobile device, collect a participant’s environmental information, and ensure the security of both the information that you transmit to the device, as well as the information that you collect from the device are all still open issues.

Today, usability testing has come a long way from being confined to a lab. As technology continues to progress and our users’ expectations of a flawless, contextual user experience grow, we must continue to explore new ways of creatively collecting data on how our applications delight or disappoint our users.

5 Comments

Great article. I would really like the author to post a follow-up piece that expands on remote usability testing on mobile.

Thank you, Rebecca, for your article!

In response to some of your cons of remote unmoderated tests—many of your points are avoidable.

  1. ‘Ascertaining the why when users confront a problem’—A skilled researcher would be able to design a test to capture these insights. You can ask focused questions during and after the session to elicit these juicy responses. For example, clearly defining success criteria in the script will prevent false reporting of task success.
  2. Non-disclosure agreements—Some companies will administer and collect NDAs for you, including UserTesting.
  3. Narrow demographics—Again, UserTesting will help you recruit your exact target market.

Note from the editor: This comment was contributed by an employee of UserTesting.

Nice article. Very spot on. I used to work at UserZoom, so I’m pretty familiar with Unmoderated Remote Usability Testing (URUT). Legal at big companies can be a pain for sure and more difficult with URUT. A work around for that can be removing your branding from the prototype. That helps sometimes because then no one will know the prototype is from you. Really concise and great article. For someone getting their feet wet, this is way better than reading a whole (and boring) book on the subject.

Thanks, Scott! Excellent point about removing the logo and any identifying marks.

Thanks, Jonathan!

Good points all. I definitely agree that a carefully crafted script is key in any kind of testing to avoid misinterpretation of the results. And certainly you can have a post-test interview to follow up on anticipated clarifications. One of the advantages of many unmoderated services today is the ability to quickly change the test and reissue it to try to get more insights in a timely fashion. However, there remains that slice of why that requires more of a real-time interaction with the user. Thanks for you comments!

Join the Discussion

Asterisks (*) indicate required information.