Testing Products People Will Use in a Variety of Environments

Ask UXmatters

Get expert answers

A column by Janet M. Six
September 25, 2017

In this edition of Ask UXmatters, our expert panel discusses how best to test designs for products that people will use in a wide variety of environments. First, the panel discusses which and how many of the expected environments you should test within, then how to simulate those environments, if necessary.

Of course, it is also important to consider the financial and time costs of testing in multiple environments. Furthermore, you must design a usability study appropriately when testing in various environments, recognizing that particular participants may complete only certain tasks within each of these environments. The panel also explores the advantages of remote testing when users’ environmental conditions are likely to have usability impacts. Finally, I describe a two-phased approach for evaluating the design of products for use in multiple environments.

Champion Advertisement
Continue Reading…

In my monthly column Ask UXmatters, our expert panel answers readers’ questions about a variety of user experience matters. To receive their answers to your question in an upcoming edition of Ask UXmatters, please send your question to: [email protected].

The following experts have contributed answers to this edition of Ask UXmatters:

  • Mark Baldino—Co-Founder at Fuzzy Math
  • Carol Barnum—Director of User Research and Founding Partner at UX Firm; author of Usability Testing Essentials: Ready, Set … Test!
  • Steven Hoober—Mobile Interaction Designer and Owner at 4ourth Mobile; author of Designing Mobile Interfaces; UXmatters columnist
  • Peter Hornsby—UX Manager at Distribution Technology; UXmatters columnist
  • Ben Ihnchak—Co-Founder at Fuzzy Math
  • Jordan Julien—Founder of Hostile Sheep Research & Design
  • Cory Lebson—Principal Consultant at Lebsontech; Past President, User Experience Professionals’ Association (UXPA); author of The UX Careers Handbook
  • Jim Ross—Principal User Experience Architect at Infragistics; UXmatters columnist
  • Janet Six—Principal at Lone Star Interaction Design; UXmatters Managing Editor and columnist

Q: When people will use a product in a wide variety of environments, what is the best way to test the prototype?—from a UXmatters reader

“Get out into the user’s environment,” Steven urges. “First, it’s great that you recognize the product will be used in the real world and that the real world isn’t just one place, but a variety of different environments. Of course, I’ve done lots of lab testing over the years, too. In fact, my office was once the observation room for a lab and my desk faces a one-way mirror. I’ve spent lots of time trying to fake or simulate real-world environments in the lab.”

“Your first step will be to figure out what you want to get out of the testing,” replies Peter. “You need to be absolutely clear about what your goals are and assess your testing methodology against those goals. Verify again and again what your user’s actual needs are and ensure that you understand enough about those needs up front.”

Is It Necessary to Test in All Expected Environments?

“The lab is not the real world, and what I’ve learned is that you simply have to get out there. The best way to test a product involves getting out to every variant of the user’s environment, in every region. Although, I find that the environment and audience are often more important than language or regional differences. So it’s important that you do the following:

  • Get out in the field—even if nearby. Get to the actual users, not their boss.
  • Observe people using the product, instead of having them watch demos during feedback sessions.
  • Simulate the product as realistically as possible.
  • But never simulate the environment. The real world is complex, and we have to learn to make products that work for people in all contexts.”

Steven also recommends that you read his Mobile Matters column, “Succeeding with Field Usability Testing and Lean Ethnography.”

“While, theoretically, it would be great if we could say, ‘Let’s test the product in all environments,’ it isn’t always practical or really necessary to do so,” replies Cory. “In fact, the main point of testing a prototype is often to identify sticky points that need fixing—and those sticky points are likely to be just as visible in a formal usability lab, an ad-hoc testing environment such as a conference room, or remote testing via a shared screen.

“However, if users will perform different activities that are specific to particular environments, this may dictate your testing different sets of tasks with different audiences or each participant getting a finite set of test scenarios. Even so, if the impact of specific kinds of distractions in different environments really is critical to understanding how people will use a product, you can either simulate those distractions or actually conduct testing in those specific environments. Of course, doing usability testing in different environments will likely result in higher testing costs, managing more complicated testing logistics, and increased labor hours.”

“Certainly, you should test the overall usability of a product. But you should also determine whether any environmental conditions would negatively impact the product’s usability,” advises Jim. “However, this doesn’t mean you would have to test in every conceivable environment. Instead, consider what environmental conditions could actually affect the use of the product.

For example, some environmental impacts on usability include bright or dark lighting conditions; a crowded, noisy environment; distractions; or shaky conditions inside a vehicle. Also, consider how often people would use the product in such conditions. Then, conduct usability testing in environments where one or more of those conditions is present. To evaluate a product’s usability without considering interference from environmental conditions, you also conduct usability testing in a more stable environment such as a home or office—if people will often use the product in those contexts.”

“In a perfect world, I would suggest performing research in as many of those different environments as possible,” recommends Ben. “But I have yet to work in a perfect world, so I suggest that you identify a handful of environments—maybe three to five—that are at different ends of the spectrum and perform research in those contexts. If the product works well in extreme environments, hopefully, it would also work well in the contexts between those extremes. However, I would continually test the product in as many different environments as possible.”

Simulating a Variety of Environments

“If various environments are likely to affect the usability of the product, you need to get out into those environments or, at least, simulate them in the lab, so you’ll know whether the product design will be effective in likely environments,” answers Carol. “For instance, if you need to know the impact of environmental noise and distractions, you can go to a public space where there are background noise and plenty of distractions.

A common location for such in-the-field testing is a coffee shop or shopping mall. However, if you can’t actually leave the building to test in a noisy environment, but work in an office building, try the lobby of your own building. Or, if you don’t have access to a noisy lobby, pipe noise into your testing facility. A case in point: In an upcoming study, we’ll be testing a set of instructions for both experienced and lay users, where their inability to follow the instructions could have potential health risks. To provide a distraction to see how well the instructions work for users in a noisy environment, we’ll have a TV playing near the participants.”

“When creating and testing a product, you’ll want to create a situation that is as close to realistic usage as possible,” responds Mark. “For example, if you were designing software for a fast-food chain, you’d want people to be able to use an app in a restaurant, on their phone, and on the go. It’s important to make sure you’re testing in a setting that’s as close to the real thing as possible.”

“Remote testing offers the benefit of allowing participants to use the product in their natural environment,” suggests Jordan. “That said, I’ve seen a lot of value in quick-and-dirty testing with colleagues and other people to whom you may have access. Some people call this coffee-shop usability testing, which is just one of many terms that refer to quick-and-dirty usability testing. While you should do testing with actual users in their own environment at some point, quick-and-dirty testing can reveal whether, in general, human beings are able to use—and find value in—a particular design solution.”

The Higher Cost of Testing in Multiple Environments

“While you’ve not specified what you mean by environments, I’m going to assume you mean physical environments rather than platforms such as mobile or desktop,” replies Peter. “The challenge you’re likely to face when you test in many environments is the higher cost of testing. Testing takes time and money, and the more environments in which you need to test, the more expensive it gets.

“Triage your environments: are they all equally important or can you prioritize one over another? Then, look at your options for testing. Do you need to create a high-fidelity prototype or can you go low fidelity all the way? I mean seriously low fidelity—for example, a walkthrough of a paper prototype might be sufficient. I’ve found that getting a solution in front of users—even a really basic solution—can help them to better understand what they want. In this regard, very basic prototypes have an edge over more sophisticated prototypes.

“Research can be expensive, but the general rule is that the further along in the design lifecycle you do testing, the more expensive testing and resolving the issues you discover during testing will be. So test early.

“Whatever approach you use, keep reflecting on what you’re doing. The more deeply you think about the problem you’re trying to solve—reflecting not just on the data, but on your research approach—the more value you’ll get from investing even limited resources in testing.”

A Two-Phased Approach

Once you’ve done your best to design a solid prototype—whether a complex, digital prototype or a simple paper prototype—it’s time to test it. First, make sure you catch all the low-hanging fruit by doing some quick usability testing, in any environment in which real users would actually use the final product. Many design flaws become very apparent as soon as real users get their hands on a product. Regardless of the environment in which particular users will actually use the product, you must fix these fundamental problems.

Subsequently, test your prototype in the most important environments. Which ones are those? To decide, use the 80/20 rule: “80% of our profits come from 20% of our users.” Do everything you can to make those users happy! Once you accomplish this, you might find that those varied environments are not quite so different as they seemed at first. For example, two environments may both be extremely noisy or have lots of distracting physical motion. So you might, in fact, be able to cover more of those environments than you originally thought. 

Product Manager at Tom Sawyer Software

Dallas/Fort Worth, Texas, USA

Janet M. SixDr. Janet M. Six helps companies design easier-to-use products within their financial, time, and technical constraints. For her research in information visualization, Janet was awarded the University of Texas at Dallas Jonsson School of Engineering Computer Science Dissertation of the Year Award. She was also awarded the prestigious IEEE Dallas Section 2003 Outstanding Young Engineer Award. Her work has appeared in the Journal of Graph Algorithms and Applications and the Kluwer International Series in Engineering and Computer Science. The proceedings of conferences on Graph Drawing, Information Visualization, and Algorithm Engineering and Experiments have also included the results of her research.  Read More

Other Columns by Janet M. Six

Other Articles on Usability Testing

New on UXmatters