Conducting Qualitative, Comparative Usability Testing

How to Conduct Qualitative, Comparative Usability Testing

Testing multiple design solutions requires you to consider a few additional factors that don’t come into play when testing a single design.

Avoid Testing Too Many Different Designs

Be wary of testing too many different designs. Testing two or three different design solutions is ideal. With more than three designs, testing becomes too complex for both you and the participants. People have difficulty remembering and comparing more than three designs. Plus, the more designs you test, the fewer task flows you can include in a study.

Test Only Designs That Actually Are Different

There should be obvious differences between the designs you test. Otherwise, what’s the purpose of testing them? If the differences are too subtle, participants may have trouble distinguishing what the differences are. So it would be awfully hard for them to compare and form opinions about those differences if they can’t readily perceive them. If designers have created designs that differ only slightly from one another, they should make an effort to come up with more varied design solutions before testing them with users.

Don’t Make Every Decision by Testing

Be mindful that you should not postpone making too many design decisions until after you’ve done qualitative, comparative usability testing. If the answer to every project team dispute or question is, “Let’s test it with users,” that’s a sign the team is overrelying on testing when making design decisions. The purpose of comparative usability testing sin’t to allow users to make every design decision. It’s about gathering information so you can evaluate and choose the best aspects of two or more designs. You should determine in advance which elements of the designs it is most important to compare.

Don’t Create an Alternative Design Just to Have Another to Test

Use qualitative, comparative usability testing only when you actually have different designs to test. Although trying out different design directions is usually ideal, sometimes there really is just one good way to design something. For example, if you’re following a common design pattern or convention, it wouldn’t make sense to do something different, unless you can radically improve on it. So, if you don’t have different designs to test, don’t force designers to create alternative designs just because you feel you should do comparative testing.

Have All Participants Use All Design Solutions

Although the quantitative data—including task-completion and error rates—that you gather is important, the qualitative data that you get from having participants think aloud and compare their experiences with each design solution is equally important. You’ll get the best qualitative data when participants can try the same tasks using multiple design solutions, then talk about the differences between them. Therefore, having all participants experience all design solutions is the best method of testing multiple designs early in a project.

In contrast, having each participant complete a task using only one design solution is more appropriate when conducting competitive usability testing, in which you compare your current user interface to those of competitors. In this type of benchmark testing, gathering quantitative metrics is most important, so you should avoid the order effects that would occur if participants performed the same tasks using different design solutions. Since participants’ qualitative opinions are less important, you don’t need each participant to experience and compare each design.

However, if participants must perform the same tasks using different solutions, be sure to alternate the order in which participants experience them to avoid order effects. For example, have Participant 1 experience solution A, B, then C; Participant 2, solution B, C, then A; and Participant 3, solution C, A, then B. This avoids the problem of tasks becoming easier with each subsequent solution they experience.

Choose the Most Important Tasks to Test

Testing multiple design solutions takes more time, so this usually limits the number of tasks and questions you can include in a study. Limit the tasks to only those that it is most important to test. For example, if you ask participants to perform the same five tasks using three different solutions, that’s actually 15 tasks. Plus, you need to save a significant amount of time at the end of each session for participants to compare and discuss the three different designs. Carefully choose the most important tasks and questions so you won’t overwhelm participants or make them rush through the session to try to fit everything in.

Gather Participants’ Opinions at the Appropriate Moments

When comparing different solutions, it’s often best to ask participants questions at different three points during testing, as follows:

• after each task
• at the end of testing each solution
• at the end of the test session, when comparing all designs

After Each Task

After participants complete each task, ask them general questions about how they felt about that task, as well as specific questions about their actions or problems they encountered. By asking these questions immediately after each task, you’ll be able to capture this information while it’s fresh in their mind.

At the End of Testing Each Solution

Once participants have completed all of the tasks using a particular design solution, ask them general questions about their overall opinions of the solution they’ve just experienced. When you get to the second and third solutions, participants will naturally start making comparisons with the previous designs, whether you ask them to or not. For example, they might say, “Oh, I like this one much better than the first one!” Regardless of whether you want them to make these comparisons at that moment, it’s best to be flexible and let them provide their opinions while they’re uppermost in their mind.

At the End of the Test Session

Once participants have completed the entire test session, ask them to look back at each design solution, compare them, and discuss their relative merits or issues. Show them each design again, to refresh their memory, and allow them to review the designs again as they answer your questions. Otherwise, it can be difficult for participants to accurately remember the differences between the various designs.

To ease your discussions, give each design solution a generic name such as Solution A, Solution B, and Solution C. At this point, you can ask participants questions about how the designs compare and their overall preferences and opinions. Gathering information about what aspects of each design they like or dislike is much more important than which solution they prefer overall. You’ll often find that there is no clear winner over all the other designs, but you can discover which aspects of each solution work well or poorly. The designer may be able to combine the best elements of each solution.

Gather Ratings in Addition to Opinions

While participants’ qualitative comments and opinions are important, they can sometimes be hard to compare. For example, a participant might give you these opinions regarding the different solutions:

• Solution A: “I like this one.”
• Solution B: “This one’s pretty good, too.”
• Solution C: “I guess this one’s not too bad.”

While you can ask additional probing questions to get more detail, how can you compare such opinions about these three solutions? And how do you compare one participant’s feedback—“It’s pretty good.”—to another participant’s—“It’s okay.”? Clearly, one comment is slightly more positive than the other, but by how much?

In addition to asking for participants’ opinions, ask them to provide ratings for the various design solutions. You could do this using a questionnaire such as the System Usability Scale (SUS) or devise your own Likert scale. Even with a small number of participants, you can compare average ratings across participants, as well as the different ratings from a single participant. For example, if a participant gave Solution A a rating of 6 out of 7 and said, “I like this one,” while she gave Solution B a rating of 4 out of 7 and said, “This one’s pretty good, too,” the numbers provide a better comparison than those two statements. Similarly, if you found that Solution A received an average rating of 4.3; Solution B, 6.1; and Solution C, 5.4; these numbers are much easier to compare than the participants’ qualitative statements.

Try Qualitative, Comparative Usability Testing

The next time you begin a project, plan to do some early usability testing on two or three different design solutions. When you plan to do that up front, you’ll find that designers spend more time exploring different design possibilities. They’ll also tend to keep an open mind longer and avoid getting too attached to a particular design solution too soon. Testing multiple designs often reveals new ideas that you’d never considered before and, thus, can help you come up with an even better design solution.