Top

Usability Testing Is Undermining UX Design

Innovating UX Practice

Inspirations from software engineering

A column by Peter Hornsby
October 20, 2014

I’d be the first to admit that there are a lot of things that irritate me. These include, but are not limited to the following:

  • people referring to a small, potent coffee as an “expresso”
  • people saying “pacific” when they mean specific
  • the use of the word intuitive in describing a design or product requirements
  • anything else that undermines the delivery of effective UX design

And although I’ve never before considered usability testing as something that falls into the large—and growing—list of things that undermine effective UX design work, I’ve recently had a number of conversations with designers that suggest their perception of usability testing is fundamentally wrong. I’ve heard both junior and senior designers express their perception of usability testing in different ways, but the core message is the same: They believe that nothing can be known about a design that a team is going to implement unless that design has been tested with the target audience. That no knowledge is possible and nothing can be said about a design with any degree of confidence, unless its usability has been validated for specific use cases, in specific circumstances, with a specific set of users, and for a specific combination of browser and device.

Bullshit.

Now, to clarify my position before the hate mail starts rolling in: I’m a big fan of usability testing. It’s a useful tool, and one that all UX designers should have in their toolbox. But we need to see usability testing in a broader context and consider its strengths and limitations.

The Role and Practicality of Usability Testing

While usability testing may take many different forms, I’m primarily going to consider the testing of a UX design solution with a representative sample of users. Of course, there are other forms of testing, but most UX professionals would consider doing perhaps two or three rounds of usability testing of a prototype in a realistic setting as meeting the need to test the design with users.

The first problem is that usability testing is expensive. If you have a testing budget, that’s great! But like any budget, you need to focus your resources where they will do the most good. Designing a really effective test that isolates the key variables that you’re evaluating, creating the prototypes necessary to test them, recruiting suitable people to participate in the testing, and analyzing the results of a study are all time-consuming tasks that require highly skilled people and typically necessitate UX designers’ taking time away from UX design tasks.

Yes, there are other types of testing available—for example, guerrilla testing and A/B testing. Guerrilla testing can be very resource efficient and offers a good balance between the time it takes and the quality of the results. A/B or multivariate testing can deliver a lot of value—if you have the development resources to create two or more versions of a feature and test them with users on a live site or application. But any form of testing takes skill and money to plan, create the necessary code, run the test, and analyze the data. When the pressure is on to deliver a product or User Experience has not yet become fully established within an organization, usability testing will more than likely take a backseat to other tasks. Of course, this is not ideal, but in reality, it happens; so it is important for UX designers to keep pushing to make usability testing part of the UX design process.

What We Know About UX Design

Now, let’s take a step back. Design is something that people have been doing for centuries, and there is a large body of relevant knowledge about UX design that stretches back for decades. I came to UX design from a human factors, or ergonomics, background—a discipline that really got started during World War II as a result of studying how people used tools to help them work and fight more effectively. The word ergonomics derives from the Greek words ergon, meaning work, and nomos, meaning laws. The term ergonomics is in wider use in Europe, while the term human factors tends to be more prevalent in the US—particularly in relation to information technology (IT). Ergonomics considers biology, psychology, the surrounding environment, and other disciplines as the foundation for design.

Coming from this background, rather than saying, “We know nothing about a design until we’ve tested it with the target user population”—to oversimplify massively!—I know that we have a solid understanding of people and how they perceive and interact with the tools they use to accomplish their work.

Let’s look at a trivial example: when designing a form, ergonomics would inform the design by considering the following:

  • the physical attributes of users—These would inform color choice and coding for accessibility by people who are physically impaired and the appropriate controls to use.
  • how people scan pages—We can draw on eyetracking research to inform the layout of forms.
  • how people complete online forms—This knowledge helps us to define the appropriate relationship between a label and the corresponding control on various devices.

More specific knowledge—perhaps in the form of personas that have derived from research with the target audience—would help inform our choice of the optimal language to use in a user interface and what Help content to provide.

My perspective on usability testing isn’t unique to UX designers who have trained in human factors, or ergonomics. From speaking to designers who have training in other disciplines—including product, vehicle, graphic, or digital design—I know that the basic principle of designing with as much of this baseline knowledge as possible—together with an understanding of a specific audience of users—plus conducting expensive usability testing to fine-tune a design is fundamental to most UX design training. Trained UX designers continually build on this fundamental knowledge by keeping up to speed on advances in UX design principles and patterns by reading articles on sites such as UXmatters, following designers on Twitter, and reading UX design blogs.

Since this is the case, why do so many UX designers play down the value of proven knowledge? It’s possible that one reason could be frustration on their part. UX designers may just be tired of arguing their case and unwilling to accept what the highest-paid person in the company is asking them to do, so may suggest deferring to the results of usability testing as a way to sidestep a potentially difficult challenge. On the other hand, there are situations when two competing design solutions may seem to have equal merit, and usability testing can be a good way to decide which one to choose.

Summary

User experience may, in some respects, be a victim of its own success. There are many articles about how to get started in UX—of course, not all are of great quality. Because everyone uses the Web, everyone has an opinion that they feel is valid. However, not everyone has learned to be objective and step back from their own experiences.

Even experienced UX designers encounter daunting challenges in managing stakeholders. They may be unwilling to defer to the highest-paid person’s opinion and hope to avoid conflict during the design process by suggesting, “We won’t really know unless we test it.” Doing this can sometimes make a designer’s life considerably easier. But the long-term cost of doing this is huge:

  • Designers may be condemning themselves to reinventing the wheel instead of innovating.
  • Users labor under working with user interfaces that fail to meet their needs.
  • Most insidious of all, stakeholders lack respect for UX designers who apparently know no more than anyone else on the team—if, after all, we don’t really know anything until we test it. 

Acknowledgments—I’d like to thank Pabini Gabriel-Petit, Janet M. Six, and Laura Eydmann Fudge for their thoughts around this topic. All errors and opinions are my own!

Director at Edgerton Riley

Reading, Berkshire, UK

Peter HornsbyPeter has been actively involved in Web design and development since 1993, working in the defense and telecommunications industries; designing a number of interactive, Web-based systems; and advising on usability. He has also worked in education, in both industry and academia, designing and delivering both classroom-based and online training. Peter is a Director at Edgerton Riley, which provides UX consultancy and research to technology firms. Peter has a PhD in software component reuse and a Bachelors degree in human factors, both from Loughborough University, in Leicestershire, UK. He has presented at international conferences and written about reuse, eLearning, and organizational design.  Read More

Other Columns by Peter Hornsby

Other Articles on Usability Testing

New on UXmatters