Overcoming Participants’ Test Mentality: Qualitative Research Approaches for Web Sites

February 18, 2013

Recently, when I received a UX brief for the evaluation of a Web site, I recognized that there was something different about this brief in comparison to others that I’ve received. This brief involved evaluating a micro-site that had been up and running in 22 markets for a few months. The site had cleverly used an advertisement as content. Visitors could click the advertisement to interact with it—and in doing so, reveal more about the company’s services. The reason that the brief was different was because it wasn’t about navigation—it was about communication.

I recognized that it would be important to remove myself from the research process to capture the realities of how the Web site communicates its services. In other words, I needed to be a fly on the wall. The solution? An online Bulletin Board that would give participants the time, freedom, and confidence to explore the site in their own time and their own environment.

Champion Advertisement
Continue Reading…

This brief highlighted for me how online, qualitative research techniques can generate valuable insights. But matching the proper qualitative research technique to a brief has become quite an art. As a qualitative researcher, I deal with lots of different research topics. But user experience is a specialist area that, perhaps more than any other area of qualitative research, really requires some thought about the best approach for each brief. Why is this? Because doing face-to-face UX research has some disadvantages that can really impact the reliability of the research findings.

I love doing UX research and specialize in it, but even so, I am still learning about the best techniques for tackling specific objectives. What have I learned so far? For the remainder of this article, I’ll share the benefit of my experience in UX research with you, highlighting the challenges that you might face in certain scenarios and the solutions that you should adopt to overcome them.

Face-to-Face Qualitative Research Methods

Face-to-face qualitative research usually involves individual, in-depth interviews, with a moderator sitting next to and shadowing a participant. Such interviews are generally useful when a site design is in a wireframe format—so the moderator may need to intervene if a participant encounters parts of the site are not yet designed—or a Web site is already built or even up and running.

Another qualitative approach is the Web clinic, which is a test involving banks of personal computers and eight or so participants using a Web site simultaneously. Web clinics are generally useful in evaluating navigation rather than Web site design and are usually conducted during the final stages of Web site development.

Qualitative research sessions often occur in a viewing studio, which uses video technology to enable observers behind a mirror or just in another room to see participants’ face and mouse movements on a screen. Sometimes, organizations set up such a studio to support video streaming across the world, allowing observers to watch and listen in their own offices.

Disadvantages of Face-to-Face Methods and How to Overcome Them

There are some specific research objectives for which face-to-face methods are most appropriate—and they usually fall under the broad areas of navigation and search. For this type of research brief, it is important to be able to observe participants’ routes through a Web site. Testing a Web site’s navigation is quite often a goal of a research brief, so we rely on face-to-face methods of research a great deal, but there are some distinct disadvantages to them.

The first disadvantage is that participants always feel like they’re being tested, even if it doesn’t show. Some participants are visibly affected by the presence of technology, cameras, and a viewing facility—if you’re using one—or even just by the presence of a moderator and her recording equipment in their home.

Plus, it is common to instruct participants to “Speak your mind as you go through the Web site,” making them feel that they are under even more pressure to perform. Unfortunately, this often results in their starting to click links fairly randomly and unnaturally. At worst, this might manifest as a series of very unusual clicks that take a participant into a very unusual area of a site that he would never visit intentionally! On occasion, I’ve stopped an interview because a participant became flustered and asked me, “Am I doing it properly?”

This is, I am sure, a natural human reaction. I’d love to be able to say that we just need to warm participants up properly. But even with the best warmup in the world, participants’ test mentality persists. The consequence is that a test’s resulting insights regarding navigation or search may be unreliable. But you can do three things to mitigate this problem to some extent:

  • Ignore participants’ early clicks. At the start of an interview, participants may be quite nervous. So, ignore their clicks early in an interview, when they’re still nervous. Once participants get into the swing of things, you’ll observe that they move around a site more naturally.
  • Assign tasks that replicate real-life scenarios. Later in an interview, assign participants tasks—which are a kind of test, I know!—that use real-life scenarios. Some example tasks might be searching for product details, comparing products, choosing a product, or going through an ecommerce payment system. Such tasks reflect real life to some extent, so relax participants.
  • Give participants control of the mouse. Ensure that participants can click whenever they need to. The mouse should be readily at hand at all times. Even extremely experienced moderators can lose track of a session if a participant feels he does not have control over the mouse.

But, if none of these approaches works for you, there is one more trick you can keep up your sleeve when moderating a session: You can ask participants to have a look around a Web site on their own for a few minutes, then leave them alone in the room for a while and watch them from behind the viewing mirror. On their own, without the presence of a moderator, participants click links quite naturally. You can use this approach very successfully at the start of an interview. It works so well that I am increasingly using it as my preferred method for testing navigation. It is especially useful if the objective of your research is to evaluate whether users’ entry into a Web site is intuitive for them.

The Importance of Observation

What I’ve described so far highlights how much the test environment and the presence of a moderator affect participants. Nevertheless, when you’re testing navigation and search, you need to monitor what’s going on, so face-to-face methods are the most reliable way of evaluating a site. But here’s the thing: moderating is more about observation than anything else, so you should be watching participants’ hands and face as much as what is happening on the screen.

Why? A participant’s eyes may sometimes follow the mouse, but not always. Occasionally, participants follow their eye movements with the mouse, pointing where they are looking, but this is not that common. People usually look at the screen while their hand remains motionless on the mouse, moving the mouse only to click a link. Therefore, observing where participants look is important. To help during analysis, you can use PiP (Picture in Picture) technology to watch video playback of both the screen and a participant simultaneously.

Closely follow participants’ route through a Web site. Note the patterns: where the user journey begins and where it falters. When a participant hesitates or falters, those observations are gold dust. It’s your responsibility to capture whether the participant is confused, irritated, or just rereading some text. People naturally read from the left, so as a rule, participants’ eyes initially focus fleetingly on the left. When people visit a previously unknown Web site, I find that their clicking links on the left is more common.

Testing systems’ processes is another good application of a face-to-face method. For example, I think it’s best to tackle testing of ecommerce systems face to face, so you can observe how intuitive a Web site’s purchasing and payment processes are. Companies lose a huge number of purchases because of poorly designed ecommerce systems. Therefore, it is important to capture information about participants’ expectations before testing proceeds. Amazon has a very well-regarded payment system that establishes the benchmark by which people measure other ecommerce systems, so don’t be surprised if many participants tell you this! Your client’s ecommerce process may well have restrictions, so capturing participants’ expectations before test sessions and observing how they manifest during the sessions lets you report the corresponding issues as category 1, 2, or 3 barriers to purchase.

Major Issues with Face-to-Face Methods

While I’ve already alluded to some of the pitfalls and weaknesses of using face-to-face methods of research, there is a big, fundamental issue regarding interviews. It is well known that participants in qualitative research want to be helpful and please the moderator. After all, you’re paying them, and it’s normal for people to want to feel they’re doing something worthwhile! But this sometimes means that they want to find problems—perhaps where none actually exist. Therefore, as a moderator, you need to be absolutely sure that any problems participants identify are things that you should report as issues. So, double-check each participant’s key problems at the end of the interview, and watch for patterns across participants.

Also, consider that you are, in effect, creating a hothouse environment for evaluating a Web site, so be careful regarding the length of an interview. If it is too long, participants may overanalyze the site. I am increasingly running interviews that are shorter in duration, so reflect the normal duration of a browsing session.

As a moderator, be aware that you are dealing with technology, and technology inherently fosters rational intercourse. But, while the interactions between humans and computers are indeed based on logic, consider that the experience itself is emotional. In other words, while a Web site may function perfectly well, the experience may be boring or lack qualities that result in engaging communication. Take the time to capture participants’ emotional reactions to a Web site through careful questioning.

Advantages of Online, Qualitative Research Techniques

If a Web site is in its finished form, so its use requires little instruction, removing the moderator from the process increases participants’ confidence and offers a much more realistic context in which to conduct research. When observation of participants and guidance from a moderator are unnecessary, online qualitative techniques such as Bulletin Boards and Moderated Blogs can really help in overcoming the disadvantages that result from the staged nature of a face-to-face research session.

Bulletin Boards and Moderated Blogs are platforms for online qualitative research. They enable a researcher to post multimedia content along with a series of questions for participants to answer. In addition to providing a text response, participants can use a range of innovative tools such as scribble pads, image annotators, and virtual mapping. Participants can also provide video feedback using a Web cam or Smart cam.

I’ve used Bulletin Boards for specific projects involving live Web sites that have already been up and running for some time and are perhaps in need of adjustment. This qualitative method is suitable for research briefs that deal with post-testing of Web site redesigns, communication, and branding. Bulletin Boards are also an excellent tool for springboarding a redesign when a Web site is in need of a refresh. However, you should not use this approach when evaluating wireframes.

The usual scenario for a Bulletin Board is to invite participants to answer a few simple questions regarding their current knowledge and experience of a Web site, then provide links to the site in question—and perhaps competitor sites, if desirable. The process continues over a period of a day or two, during which the researcher has an open dialogue with each participant and can post probing questions.

There are some great advantages to using Bulletin Boards and Moderated Blogs, as follows:

  • Participants can view a Web site on their own time and at their own pace—eliminating the issue of creating a hothouse environment.
  • Participants use their own computer, tablet, or smartphone, so these approaches replicate the way in which they would typically experience a Web site at home, at work, or on the go.
  • Working in a familiar environment means participants treat a research session less like a test and more like a normal browsing experience.
  • In the case of a Bulletin Board, you can choose to let participants view each other’s responses—if you fees this is wise. This can give you insights into a group view of what a site’s core issues are, especially from an emotional standpoint.

Issues with Online Research Techniques

There are fewer disadvantages to doing research using Bulletin Boards and Moderated Blogs—assuming you’re applying these techniques to the right objectives. But there are one or two issues for which you should be prepared:

  • Pay close attention to the phrasing of your questions because participants may misinterpret written questions. It’s always a good idea to pilot your discussion guide before going live.
  • If participants don’t write much in response to your questions, you may be able to encourage a fuller response through further questioning. However, I’ve found that, unless participants are especially passionate about a particular issue, they won’t have a lot to write about it. This can, in fact, be an advantage, if you consider my earlier point about participants’ claiming a Web site has problems that don’t actually exist.
  • Participants may use incompatible hardware—for example, an iPad that cannot display Flash—so be clear about any hardware requirements when you are recruiting.
  • Finally, if there are particular pages or aspects of a Web site on which you want to focus, it is wise to take screenshots and post them on the Bulletin Board. This helps avoid any confusion. Most online, qualitative research software includes an image annotator that lets participants add notes to an image to which they’re referring in a discussion.

Remote Interviews

Remote interviews are another technique that I’ve used from time to time. They require participants to install software on their computer that lets you monitor their actions remotely as they move through a Web site. During a remote interview, the moderator and participant converse using headsets. The only real advantages of this method are cost savings and the benefits of broad geographic coverage, and remote interviews suffer from the same problems as face-to-face interviews: the moderator’s presence and participants’ test mentality. Plus, there is usually a time lag between voice communications and visuals, which can mean you lose vital details.


Eyetracking can supplement traditional qualitative research approaches. It is becoming more affordable and more widely available as equipment makers improve their technology. Thus, it is slowly moving out of labs and universities into mainstream use by agencies. Over the next year, as I learn more about this technique, I intend to incorporate eyetracking into my plans for UX research projects—but only when the brief calls for it!

Some Final Thoughts

Qualitative UX research is becoming a very specialized area of research. You must carefully consider what would be the right approach for a particular study, as well as how face-to-face research might affect participants’ feelings and responses and, thus, the results of your research.

Web site owners have become very accustomed to watching participants using their site through a window in a viewing facility and witnessing first-hand their usage of and reactions to the site. However, face-to-face research has many disadvantages, so as a qualitative researcher, you should invite your clients to consider newer, online approaches to research that do not have those issues.

Ultimately, there is no single perfect research solution to a client brief, but there are many steps that you can take to lessen the disadvantages of either online or face-to-face methods of research. 

Chief Technology Officer at Industrial Internet Consortium

London, England, UK

Steve MellorSteve has 15 years of qualitative research experience with research agencies, including Harris Interactive, RDSi, and Tpoll. As a qualitative researcher, he has worked on some of the world’s largest Web sites, including Disney, PepsiCo, and Nespresso. He has completed many hundreds of interviews with adults and children to improve the usefulness and usability of these Web sites. Steve is a frequent speaker at Market Research Society (MRS) conferences in the UK and tutors candidates for the MRS Advanced Certificate in Market Research. He is a member of the Association of Qualitative Research (ARQ), MRS, and the Advertising Planning Group (APG).  Read More

Other Articles on User Research

New on UXmatters