User Experience and Scientific Methods, Part 1: Eyetracking
Published: August 5, 2013
In this series, I’ll describe the application of some of the scientific techniques that user experience and market research have adopted, including eyetracking, EEG, and fMRI. Based on my experience with these techniques, I’ll debunk some myths surrounding their use. This series will help you to understand
- their technical limitations
- fundamental limitations attending their use within the UX industry
- implications of introducing a new scientific method to our industry
I want to ask you to do two things while reading these articles. The first is to maintain an open mind. The second is to take a step back and really think about what happens when a sensitive and specialized scientific tool becomes commoditized.
Ruthless Profiteering Breeds Bad User Experiences
One element of remaining competitive is to develop a unique selling proposition (USP)—for example, adopting a scientific technique in the industry of user experience. For this to work, adopters must be experienced in using the tool. Their knowledge has to be strong enough to enable them to adapt its function and apply it effectively to a new area.
Naturally, this forces others to either catch up or differentiate in other ways. Those playing catch-up may have little experience with the technique, so are unable to make an informed decision about how to apply it. As a consequence, they are under pressure to sell and use something that
- they don’t understand
- has ambiguous value
- has hidden limitations
The more obscure an approach seems, the harder it is to understand when you should use it, when you could use it, and when it’s a complete waste of time.
Those who don’t really understand a technique often try to sell it at every opportunity, believing that it always adds value. This devalues the technique’s strengths and overcomplicates what should often involve a very simple form of descriptive testing and discourse.
The more we use something, the less special it becomes. As we begin to accept its use as providing a minimum level of value, this undermines a technique’s actual added value. We can often attribute this to a lack of transparency around how the technique works. Often, the pioneers who make use of a service seek only to distance themselves from the competition. This promotes short-term company growth, but hinders industry development. Many who could clarify the situation do not want to set themselves apart in their field as leaders or educators.
All of this forces other UX professionals to guess when to use a tool or technique, with the result that its real value and applicability become obscured, and many come to perceive it as being right for any job.
Eyetracking and User Experience
The most obvious scientific technique to achieve a high rate of adoption in the UX community is eyetracking. However, the average UX professional’s knowledge of its scientific applications has remained focused on how and when to apply eyetracking—not why.
As a result, eyetracking guides are big business, and many great, practical application guides have been created. However, few have taken the time to step back and look at the impact that false claims and the forceful introduction of the technique have had on the UX industry.
We No Longer Sell Clients on Eyetracking—They Demand It
One of the hardest parts of our job can be justifying a new or novel approach. This is especially difficult when we begin to work with a new client who has experience working with one or more agencies—especially when that agency views eyetracking as a minimum standard for research.
Clients are now requesting eyetracking without any consideration or awareness of the true implications of including eyetracking in user research. And why should they? The majority of our clients have little or no knowledge of the technique’s scientific heritage. They request it because agencies have sold eyetracking to them as the gold standard in research rather than educating them about the situations in which its use is appropriate. They perceive it as a bar that their research must meet and as being ubiquitous in usability testing—not as an ancillary service.
Profiteering on Eyetracking
Agencies frequently drive the hard sell by pushing eyetracking as an essential addition to user research. Many of them don’t fully understand what the tool can and can’t do, so include it in order to cover all the bases. They don’t offer their clients a baseline measure for a true standard in usability testing. Nor do they provide an informed comparison of the quality versus the quantity of their included services.
Once clients have been sold on using eyetracking, we must persuade them that it’s not always necessary to use eyetracking. We must reassure them that it’s okay to go back to the old-fashioned style of usability testing—without the bells, whistles, and tinsel. We must show greater reliance on something in which we are experts: discourse.
Using Eyetracking: Science Versus User Experience
To fully appreciate why it is okay to rely on more natural forms of investigation, it is important to understand how scientists actually use eyetracking—and why the UX industry is physically incapable of using eyetracking in the most appropriate way.
Reason 1: User experience uses the least invasive, but also the least accurate eyetracking hardware on the market.
At the University of Kent where I studied, the eyetracking hardware that they use varies, depending on participants’ needs. Most agencies use a Tobii T60/T120. The uninvasive nature of the Tobii means that it’s incredibly forgiving of head movements. As a result, it provides the only way to conduct eyetracking experiments using children. However, in a scientific setting, researchers would not consider using this tool for testing with adults. It is simply not accurate enough. Instead, they would use a head-mounted device. But this would never work for a usability study because it would make participants feel uncomfortable—both physically and socially.
Reason 2: Eyetracking wasn’t designed for usability studies!
The reason that scientific studies use a head-mounted device is because, within reason, it doesn’t matter how comfortable or uncomfortable the participants are. Eyetracking was originally designed to evaluate subconscious biases—something over which people have little or no control. In science, eyetracking operates in a context-free environment. Scientists reduce all external influences to measure innate, invisible, and uncontrollable cognitive processes. Science relies on refutation, removing as many variances as possible and working from the bottom up.
Reason 3: Eyetracking doesn’t track what we see.
The key issue with eyetracking is that the average UX professional has little knowledge of how people process information. Tracking a person’s gaze can’t inform us about their conceptions or true perceptions of information. The brain handles that unconsciously. Before we start paying attention to something, we have to work out whether it’s worth our time.
The brain has an internal moderator that manages where we look and the information that we take in. The amount of information available to us at any given point in time is overwhelming. We simply cannot attend to all of it. Therefore, we use a set of strategies that ensure we attend only to important information. To work out what to take in, we have to work out what we can take in. That’s where discourse steps in—exploring what participants actually read, what they understood, and what they now know.
Reason 4: Eyetracking is a numbers game.
The more elements that you add to a stimulus, the more likely it is that someone will look at them. When using eyetracking to test Web sites, the opportunities for participants to change their gaze shoot up exponentially. It becomes harder for us to understand what they have truly looked at and where they have merely glanced. To get reliable results, we have to use more participants.
Eyetracking is a quantitative technique. It relies on high numbers of participants to establish an effect. The more people who attend to an area, the more certain we can be of its stealing their gaze. Usability testing is a qualitative technique. It pays less attention to numbers and instead focuses on rich, informative data that is based on opinion. This forces us to shoehorn eyetracking into a qualitative context. It can also shift the focus of a study. It’s easy to fall into the trap of basing your entire analysis on the eyetracking results, while losing track of the strengths that come from a dominantly discursive approach.
When combining qualitative and quantitative approaches, your must sacrifice one of them. You can’t easily and cost effectively get rich, qualitative data from hoards of participants, and you can’t get reliable, valid quantitative data from just a few participants.
User Experience Appears Immature as an Industry
Because we can’t easily give eyetracking the numbers of participants that it requires, the results that we get from eyetracking are fundamentally limited. They cannot establish a direct link to cause, but they can
- act as a strong focal point for discussion
- provide a way for good facilitators to focus the dialogue
- highlight where a design is letting users down
Eyetracking’s introduction to user experience has forced many agencies to offer the service—regardless of whether they believe in it—so many clients now view an eyetracking service as a measure of competency. False promises around what eyetracking can and cannot show, coupled with a lack of training and expertise on the part of researchers, may sour clients on the usefulness of the eyetracking to user experience.
Whenever we introduce a new technique without justifying its introduction and supporting its application, we reduce the quality of the results that the UX industry produces. We have always prided ourselves on being an industry that collaborates, so our communication about the use of complex scientific methods like eyetracking should reflect this.