Eyetracking: Is It Worth It?

By Jim Ross

Published: October 19, 2009

“Seeing where people look while using your Web site, Web application, or software product sounds like an opportunity to get amazing insights into their user experience.”

It is easy to get excited about eyetracking. Seeing where people look while using your Web site, Web application, or software product sounds like an opportunity to get amazing insights into their user experience. But eyetracking is expensive and requires extra effort and specialized knowledge. The heat maps and other visualizations certainly look impressive, but what can you really learn from them? After using eyetracking for the first time, many find that it is not easy to know how to analyze the visualizations and make conclusions from them. Does eyetracking really provide any additional insights you would not have discovered anyway through traditional usability testing? Does the value of eyetracking outweigh its limitations? This article will discuss and answer these questions.

What Does Eyetracking Show Us?

Eyetracking detects where a person’s fovea fixate and the movements in between fixations. The fovea is a small spot on the retina that is responsible for our fine, detailed vision. Outside the fovea, visual acuity decreases greatly. Our eyes constantly move in rapid bursts called saccades, with brief stops called fixations, during which we take in visual information through our fovea. We use our parafovea—the area just outside the fovea—and peripheral vision to determine where to fixate next.

“Most modern eyetracking systems unobtrusively track participants’ eyes by reflecting near-infrared light off the eyes and detecting the position of those reflections to determine where the person’s fovea are fixated.”

We can see these fixation points and saccades most clearly in eyetracking gaze plots. Figure 1 is a gaze plot of one study participant’s fixations and saccades during a task. The circles are the fixation points, where the participant’s fovea stopped briefly to take in information. The numbers indicate the order of fixations, and the size of each circle indicates the length of a fixation, with larger circles signifying longer fixations. The lines connecting the circles are the saccades, the movements between fixations.

Most modern eyetracking systems unobtrusively track participants’ eyes by reflecting near-infrared light off the eyes and detecting the position of those reflections to determine where the person’s fovea are fixated.

Figure 1—An eyetracking gaze plot for one participant

Gaze plot

What Value Does Eyetracking Provide?

This may all sound pretty impressive, but what value can we really get from knowing where a person’s fovea fixated?

Helping to Identify Usability Problems

“We should not use eyetracking data alone, but in conjunction with careful observation of participant behavior and discussion.”

Eyetracking provides additional information to help you find and interpret design and usability problems. It is important to note that we should not use eyetracking data alone, but in conjunction with careful observation of participant behavior and discussion. Within this context, seeing where participants looked during a task provides additional insights into what they were doing and why.

Knowing where participants looked during a task is especially helpful in answering certain types of questions. For example, eyetracking data can help you determine the following:

  • why participants had problems performing a task
  • where participants expected to find certain elements
  • whether participants noticed a particular element—such as a link, button, advertising, or something new added to a user interface
  • whether elements are distracting in a negative way
  • how efficiently a design guides participants through a task
  • whether there are differences in task performance by user group—for example, between new users and experienced users
  • which content participants read—including articles, instructions, contextual Help, and error messages
  • how participants read—in detail or by scanning
  • whether a particular design is more effective than another—in terms of user or business goals

For example, Figures 2 and 3 show an example of how a feature that its designers intended to be a helpful aid was actually a negative distraction. While filling out an online credit card application, participants were distracted by a Talk to a Specialist box, shown in Figure 2, that appeared and followed them down the right side of the page. The gaze plot in Figure 3 shows one typical participant’s fixation points (the circles) as she filled out the form. We can clearly see the three times when the chat box distracted her attention from the form. Without eyetracking, we would have observed only the third of these, when she finally closed the chat box. We would not have seen the previous two instances of distraction. The client had heard complaints about this chat box before, but it took eyetracking to show them how distracting it really was.

Figure 2—A distracting Talk to a Specialist chat box

Distracting chat box

Figure 3—Three points at which the Talk to a Specialist box distracted the user

Three distractions

Figure 4 is an example of how eyetracking can show whether a design guides participants effectively through an information seeking task—in this case, finding the dates of the beach volleyball finals on the 2008 Beijing Olympics schedule. The gaze plot in Figure 5 shows the common and extended visual paths of multiple participants as they tried to find the dates on this confusing schedule. Clearly, the design of this schedule needed major improvements.

Figure 4—Beijing 2008 Olympics beach volleyball schedule

Schedule

Figure 5—Gaze plot for the Beijing 2008 Olympics beach volleyball schedule

Gaze plot for schedule

Showing Hard-to-Articulate Behaviors

“Eyetracking can help us identify behaviors that participants cannot articulate.”

Eyetracking can help us identify behaviors that participants cannot articulate. If you ask participants whether they noticed a particular element or why they had a problem, they often have a hard time accurately and reliably answering such questions. We make many of our eye movements unconsciously, so it is difficult to report what we looked at, for how long, or in what order. Having participants think aloud during tasks or retrospectively after completing tasks provides valuable insights into their thought processes and behavior, but these are consciously filtered explanations. Eyetracking shows you participants’ actual eye movements, without any conscious filtering.

Providing Additional Insights

In addition to the data eyetracking provides for later analysis, watching the live view of the eyetracking data during usability testing provides a facilitator with additional insights—like a sixth sense—into what a participant is doing. This additional understanding helps the facilitator ask more relevant questions and, thus, get more insightful information.

Visualizing Data for Observers

“The live view of the eyetracking data provides a great visualization for observers during usability testing.”

The live view of the eyetracking data provides a great visualization for observers during usability testing. Observers can see the live view of the eyetracking data, which makes it much easier for them to follow and understand what is happening during testing. Seeing a participants’ eye movements provides much more information that just seeing a mouse pointer moving around the screen. Because eyetracking makes observing a usability test much more interesting and engaging, observers tend to pay closer attention and get more value out of observing the tests.

Exciting Clients and Team Members

Eyetracking studies excite clients and project team members. Observing live eyetracking and viewing the resulting visualizations is exciting. Eyetracking gets clients and project team members interested in observing more tests and doing more usability testing. That, in itself, is a good thing!

Providing Compelling Visualizations of Usability Problems

“Eyetracking provides compelling, obvious visualizations of usability problems.”

Eyetracking provides compelling, obvious visualizations of usability problems. Eyetracking visualizations are often a great way of demonstrating usability problems. You can describe usability problems much more powerfully—and in a way that’s visually obvious—with a good eyetracking visualization than in words alone. For example, the following sentence is much more salient when a gaze plot accompanies it: Participants had a difficult time finding the link to register for access to their auto insurance policy. The gaze plot in Figure 6 shows the extensive path a typical participant took while looking for the Register link.

Figure 6—Gaze plot of a participant searching extensively for a Register link

Gaze plot of participant looking for link

Is Eyetracking Really Necessary?

“Participants remembered 70% of the elements they had actually seen, but were not very good at remembering the order in which they had looked at those elements.”

Do you really learn anything from eyetracking that you could not learn through regular usability testing? Can a good designer foresee how users would view a design and guide their eyes through the design?

Johansen and Hansen sought to answer these questions in their 2006 study, “Do We Need Eye Trackers to Tell Where People Look?” [1] For this study, which compared conventional usability testing with eyetracking, they conducted a usability test in which participants reported from memory what they saw. They found participants remembered 70% of the elements they had actually seen, but were not very good at remembering the order in which they had looked at those elements.

They also asked experienced Web designers to predict the eye movements of a typical user and found the designers could predict only 46% of the elements users typically saw. Plus, only 36% were successful in predicting the first three areas of the screen participants looked at. The study concluded that designers cannot accurately predict where people will look, and participants cannot reliably report where they looked.

Of course, you can conduct usability testing without eyetracking, but eyetracking provides valuable additional data about where participants looked during tasks. This is data a facilitator cannot otherwise observe and participants cannot report accurately.

What Are the Limitations of Eyetracking?

Although seeing where participants fixate is valuable, there are limitations to what eyetracking can reveal.

Eyetracking Cannot Track Our Peripheral Vision

“Eyetracking can be misleading, because it does not capture peripheral vision.”

Eyetracking can be misleading, because it does not capture peripheral vision. Eyetracking records and displays foveal fixations, in the small part of our visual field that produces the sharpest vision. It does not record peripheral vision, which makes up 98% of our visual field. This is significant, because we use peripheral vision to choose where to fixate our fovea next.

So we can see things on a screen without directly fixating on them. For example, we can see banner ads on the right side of a page, using our peripheral vision. We know enough from their position on the right side of the page, their visual appearance, and from our previous Web experience to identify them as ads, and we often choose not to fixate on them.

Because eyetracking visualizations do not show peripheral vision, they can be misleading. For example, in a heat map, some assume users did not see areas without hot spots at all. It is important to recognize that the eyetracking fixations, or hot spots, do not represent everything participants saw. They are just the areas participants fixated on, viewing them in detail.

For example, Figure 7 shows two versions of the same heat map. The gaze opacity heat map on the right shows only the areas of the page with hot spots. The rest of the page is black. Obviously, this is not how we really see Web pages. We see entire pages through both our foveal and peripheral vision. The hot spots are the areas on which we fixate our fovea to see them in more detail. It is important to explain this distinction to the audience for your testing results.

Figure 7—Corresponding heat and gaze opacity heat maps

Heat and gaze opacity heat maps

Fixations Don’t Represent Attention or Understanding

“Participants’ fixations do not necessarily equal attention or understanding.”

Participants’ fixations do not necessarily equal attention or understanding. Eyetracking shows only fixation points—that is, points where the fovea stop to take in information—and saccades—eye movements between fixations. It does not reveal anything about the higher-level processes of attention and comprehension. Just because a person’s eyes fixate on something, that does not mean he or she is consciously paying attention to that element—or understands it.

This is why eyetracking data has little meaning by itself. Careful observation of tasks and discussion with participants is necessary to understand their mental processes. Eyetracking provides complementary data to aid in this understanding. It is not sufficient alone.

Fixations Don’t Communicate Meaning

“Eyetracking shows only where participants fixated, not the meaning of their fixations.”

Eyetracking shows only where participants fixated, not the meaning of their fixations—that is, it shows where, but not why. For example, if an element received many fixations or longer fixations, it could indicate either attraction to that element or difficulty in understanding it. Again, this is why you should not use eyetracking data alone. It requires interpretation. Careful observation and discussion with participants let you understand the context for your eyetracking data.

How Does Eyetracking Affect the Testing Process?

Eyetracking is not just software and hardware that you turn on to record usability test sessions. Using it effectively requires making changes to your usability testing process.

Interactions Between Facilitator and Participant Change

“Getting accurate eyetracking data that shows what participants look at during tasks requires letting participants perform the tasks without interruption.”

Eyetracking changes the interactions between the facilitator and participants. Getting accurate eyetracking data that shows what participants look at during tasks requires letting participants perform the tasks without interruption. Asking participants to think aloud, asking probing questions, or even allowing participants to ask questions interferes with capturing accurate eyetracking data.

Instead of using a concurrent think-aloud technique, you can use a retrospective think-aloud technique, in which you ask participants to describe what they were doing while you play back a recording of the tasks they performed. However, when using this technique, participants’ comments lose some of their immediacy. You should consider the trade-off between having participants think aloud and getting accurate eyetracking data, depending on which is more appropriate for the goals of your test.

Eyetracking Can Be Intrusive

Eyetracking can interfere with the naturalness of a test session. The ideal eyetracking system is completely unobtrusive to participants. Eyetracking systems have evolved from very intrusive systems that involved participants’ wearing headgear and having their heads immobilized to almost completely unobtrusive systems that look like a regular computer monitor. With these modern systems, most participants have no idea they are sitting in front of an eyetracker until they are told so at the beginning of a test session.

“Eyetracking can interfere with the naturalness of a test session.”

Yet, even with the most unobtrusive eyetracking systems, participants can experience a sense of self-consciousness—an uncomfortable awareness of everything they look at—because they know they are being eyetracked. Eyetracking self-consciousness tends to fade over time as participants get involved in the test tasks.

Another limitation of eyetracking systems is that participants need to remain within range of the eyetracker. Though modern systems can track participants’ eyes over a wide range, participants often move out of range over time—for example, they slump, lean back, lean forward, or rest their chin on a hand. When this occurs, the facilitator must ask a participant to move back into range. Each time this happens, it interrupts the participant’s task and disrupts the naturalness of the situation. If a participant repeatedly moves out of range, the facilitator has to decide whether it is more important to get the eyetracking data, or it is better to sacrifice the eyetracking data to see the participant’s natural, uninterrupted behavior.

Eyetracking Tests Take More Time

“Planning and setting up an eyetracking study can add some extra time to the usability testing process.”

Planning and setting up an eyetracking study can add some extra time to the usability testing process. Both the need to explain eyetracking to participants and the need to have participants retrospectively think aloud to the playback of the eyetracking recording either add time to test sessions or reduce the number of tasks you can test. Eyetracking also generates additional data, adding significant time to the data analysis phase. With all of these considerations, eyetracking is probably not the best choice for testing with a low budget or when you need a quick turnaround.

Eyetracking Is Expensive

Although the cost of eyetracking hardware and software has come down, it is still somewhat expensive. The cost of a new eyetracking system is $40,000 to $60,000. Rental prices can be up to $3000 per day.

Eyetracking Can Be Difficult to Learn

“Without careful planning, eyetracking results can be meaningless—and even when everything goes right, they can be difficult to interpret.”

Eyetracking is not something a usability professional can start using effectively right away. More important than learning how to use the hardware and software is learning how to effectively plan and conduct eyetracking studies and analyze eyetracking results. This is not easy. Beginners often learn through trial and error that, without careful planning, eyetracking results can be meaningless—and even when everything goes right, they can be difficult to interpret. Like anything else, analysis of eyetracking results is a skill that you must develop through experience.

Eyetracking Is Subject to Technical Problems

The more technology you have, the more can go wrong. Although eyetracking technology is continually improving in its reliability, anyone who had been involved in eyetracking has horror stories of technical problems that threatened to derail their testing. Experiencing such problems during a test session—especially with clients observing—is not fun. It can make you yearn for the good old days of old-fashioned usability testing.

Eyetracking Can Become a Gimmick

Eyetracking can be very impressive and exciting for clients, which makes them eager to do more usability testing and eyetracking. But the downside is that they may want to use eyetracking for everything, even when it’s not the most appropriate method for your testing goals. It is important to be sure eyetracking does not become a sales gimmick. Use it purposefully, where it provides value.

So Is Eyetracking Worth It?

“If you know how to use eyetracking effectively, it can provide additional insights to usability testing that can help you find problems and answer questions about user behavior.”

Considering the advantages and disadvantages of eyetracking, is it worth it? Some have concluded that the benefits of eyetracking are not worth the high cost, effort, and complexity it adds to usability testing. On the other hand, some eyetracking vendors and consultants have promoted the idea that you cannot conduct usability testing effectively without eyetracking.

The truth lies somewhere between these extremes. If you know how to use eyetracking effectively, it can provide additional insights to usability testing that can help you find problems and answer questions about user behavior. Eyetracking is not essential to usability testing, but if you can afford it and have the time to learn how to use it effectively, it is definitely worth it. Ultimately, you have to make the decision—based on your testing goals and the considerations I’ve outlined in this article—whether eyetracking would provide value and its benefits would outweigh its limitations.

Reference

[1] Johansen, Sune A., and John P. Hansen. “Do We Need Eye Trackers to Tell Where People Look? Conference on Human Factors in Computing Systems: CHI ’06 Extended Abstracts on Human Factors in Computing Systems, 2006. Retrieved October 11, 2009.

2 Comments

Thank you, Jim, for sharing this. I agree with you. We shouldn’t use eyetracking data alone. Observation and in-depth interviews will give us other parts of the whole picture.

Would you consider doing a professional usability study without capturing a screen recording? I doubt it, eyetracking is an essential tool for doing Web usability studies, just in the same way that a screen recording is, but with added value.

As an industry, we need to get over eyetracking and accept it as an essential tool in the kit bag of usability professionals. Yes, eyetracking does require additional skills, a greater level of training, competence, and more complexity, but being a great usability professional is not simple, nor should it be!

Read my blog post: “Why I�m Not in Love with Eye Tracking”

Join the Discussion

Asterisks (*) indicate required information.