UX Concepts and Practices That We Wish Would Just Go Away!

By Janet M. Six

Published: August 19, 2013

Send your questions to Ask UXmatters and get answers from some of the top professionals in UX.

In this edition of Ask UXmatters, our experts discuss some concepts and practices that defy UX best practices and, thus, have negative consequences—concepts and practices that they wish would simply go away.

Every month in Ask UXmatters, our panel of UX experts answers our readers’ questions about a variety of user experience matters. To get answers to your own questions about UX strategy, design, user research, or any other topic of interest to UX professionals in an upcoming edition of Ask UXmatters, please send your questions to: ask.uxmatters@uxmatters.com.

The following experts have contributed answers to this edition of Ask UXmatters:

  • Carol Barnum—Director of User Research and Founding Partner at UX Firm; author of Usability Testing Essentials: Ready, Set… Test!
  • Ronnie Battista—UX Practice Lead at Slalom Consulting
  • Steve Baty—Principal of Meld Studios; President of IxDA; UXmatters columnist
  • Drew Davidson—Senior Experience Director at ÄKTA
  • Jessica Enders—Principal at Formulate Information Design
  • Leo Frishberg—Product Design Manager at Intel Corporation
  • Michael Griffith—Creative Director at Bottle Rocket Apps
  • Steven Hoober—Mobile Interaction Designer and Owner at 4ourth Mobile; author of Designing Mobile Interfaces; UXmatters columnist
  • Adrian Howard—Generalizing Specialist in Agile/UX
  • Caroline Jarrett—Owner and Director at Effortmark Limited; UXmatters columnist
  • Jordan Julien—Independent Experience Strategy Consultant
  • Shane McWhorter—Executive Director, User Experience Strategy & Design at Product Concept, Design, and Experience
  • Jim Ross—Principal of Design Research at Electronic Ink; UXmatters columnist
  • Whitney Quesenbery—Principal Consultant at Whitney Interactive Design; Past-President, Usability Professionals’ Association (UPA); Fellow, Society for Technical Communications (STC); UXmatters columnist
  • Baruch Sachs—Senior Director of Human Factors Design at Pegasystems; UXmatters columnist
  • Janet Six—Principal at Lone Star Interaction Design; UXmatters Managing Editor and columnist

Q: What's a UX concept or practice that you wish would just go away? And why?—from a UXmatters reader

One True Way

“I wish that the idea that there is just one true way to practice user experience would go away.”—Whitney Quesenbery

“I wish that the idea that there is just one true way to practice user experience would go away,” replies Whitney. “We joke that, as UX designers, the answer to everything is ‘it depends’—but to some extent, it really does. We work in many different business environments and team structures. We work on different technical platforms, different devices, and most of all, for different audiences who are in different contexts and have different goals and activities. Sometimes we know an environment well; sometimes we are exploring new territory. Every practice or research method gives us answers. The hard part is deciding what the question is at any moment, on any project.

“Maybe that’s not the kind of answer you are looking for, but I find myself less interested in debating which usability method or design process is best and more interested in asking when and why it would be useful.”

“One of the most valuable things you can learn about a tool, technique, or method is not just how to use it well, but when to use it at all.”—Steve Baty

“I would like to see the notion that any concept or idea is universally applicable go away,” says Steve Baty. “One of the most valuable things you can learn about a tool, technique, or method is not just how to use it well, but when to use it at all. Well-meaning and talented people achieve mediocrity every day by applying the wrong tool to a given situation. Whether it’s wasting time on wireframes when a working prototype would be better, hand-crafting personas for a team that’s already in tune with their customers, or adopting a mobile-first approach for an internal, enterprise application that will never run on a mobile device, there’s no end of waste out there arising from the poor choice of tools.”

“I don’t think there’s any UX practice that I’d like to completely disappear. Every one that I can think of has some context where it works well,” answers Adrian. “What I’d like to see vanish is the idea of One True Way of doing user experience.

“Look for opportunities. Look for waste. Find better ways of doing things.”—Adrian Howard

“We need to look at our teammates and processes with the same eye that we use when observing the people for whom we design products. Look for opportunities. Look for waste. Find better ways of doing things. I’m bored with discussing whether personas are the best thing since sliced bread or a pointless waste of time. I’m bored with discussing whether lean UX is fantastic and new, what everybody has been doing forever, or antithetical to great design. I’m much more interested in our figuring out the contexts in which practices help or hinder. That’s a harder discussion to have, but would be much more productive.”

Expecting Perfection and Pixel-Perfect Design

“We cannot understand the motivations of every user, and all of this happens in a real and messy world. We have to begin designing for mistakes, uncertainty, and imperfection.”—Steven Hoober

“I have lots of specific objections to a focus on pixels, to a mindset that discourages design approaches from moving to other platforms, and to designers and developers saying nasty things about platforms that don’t provide adequate tools to support their visions of what makes a perfectly predictable platform,” exclaims Steven Hoober.

“But most of all—because this viewpoint hides a deeper issue—I reject the idea that there would be no problems if we just had perfect control of a product; always knew what users were doing, what device they were using, and their location; and we could always make a design that we’ve specified appear perfectly on any screen. I reject the idea that we could ever achieve 100% check-out rates for shopping carts or that everyone would have perfect satisfaction with the results of any process.

“It’s time to acknowledge that is never going to happen. That it never can happen in the universe we all live in. Our products are too complex. We cannot understand the motivations of every user, and all of this happens in a real and messy world. We have to begin designing for mistakes, uncertainty, and imperfection.”

Imitating Other Companies

“The UX practice that I wish would go away is the imitation of other companies’ user experiences that have nothing to do with your own.”—Baruch Sachs

“The UX practice that I wish would go away is the imitation of other companies’ user experiences that have nothing to do with your own,” asserts Baruch. “I hear on a daily basis from customers who tell me, ‘We want our site or application or product to be the next Apple or Google.’ The problem is that their companies are neither Apple nor Google. They have their own strengths and lines of business that do not make such imitation feasible. Instead of focusing on their strengths and differentiators, they focus on a complex game of UX oneupmanship. They want their user experiences to simply copy the features and functionality of their competitors’ products. What that does is reduce design solutions to a level of sameness that inhibits creativity and the ability to lead the pack rather than follow it.

“As UX professionals, we have to take responsibility for creating experiences that are the most amazing experiences people could have when interacting with our site or application or product. To get there, we have to do the hard work of creating rather than imitating.”

Lean UX

“I think lean UX is definitely a mixed blessing. … This philosophy aligns beautifully with creating great user experiences. But the danger surrounds what it means to test efficiently.”—Jessica Enders

“I might not say that I wish it would just go away, but I think lean UX is definitely a mixed blessing,” remarks Jessica. “The key original concept of lean is about efficiently testing assumptions and approaches, then iterating in response to what you learn. (See ‘The Lean Startup Methodology.’) This philosophy aligns beautifully with creating great user experiences. But the danger surrounds what it means to test efficiently.

“I’d wager that UX professionals who conduct quality research with users are in the minority. As Anders Ramsay says in his great post ‘Agile UX vs Lean UX—How They’re Different and Why It Matters for UX Designers,’ ‘Many UX designers give lip-service to the idea of user-centered design, but don’t actually spend a lot of face time with real users.’ On the one hand, lean UX—see Jeff Gothelf’s presentation ‘Lean UX: Getting Out of the Deliverables Business—undoubtedly encourages some UX professionals to ‘get out of the building’—see Steve Blank’s post ‘Ardent 2: Get Out of My Building.’ But on the other hand, many probably use some pretty flaky research practices.

“For instance, ‘grabbing whoever walks by’ does not a quality sample make. I used to work as a survey methodologist at the Australian Bureau of Statistics, and this was mildly termed a convenience sample and promptly relegated to the pile labeled ‘Never use for anything remotely important.’ If you think your user research method doesn’t really matter, try building an application for miners and test its use only above ground; leaving people with low literacy out of the error analysis for a form; or running trials on a public transport Web site only during school holidays. You’ll find out the hard way that when it comes to user research, efficiency doesn’t make up for poor quality.

“So, by all means, get to know lean UX and draw from that approach when it’s appropriate for your context. But remember, there is no substitute for thoughtful, rigorous design processes—which can still be fast and cheeky! For ideas on how to achieve that, read Cennydd Bowles and James Box’s book, Undercover User Experience Design.”

Focus Groups and Preconceived Notions

“I wish focus groups would go away. Oftentimes, focus groups stifle innovation…. … Focus groups may be okay for driving small incremental changes, but innovation happens in leaps. I have seen many great ideas and innovations killed in a focus group.”—Michael Griffith

“Although focus groups are not necessarily a UX practice, people often use them to gather information from users,” responds Michael. “I wish focus groups would go away. Oftentimes, focus groups stifle innovation and collect opinions that users would never normally have. Finding a facilitator who does not lead participants and finding participants who don’t suffer from group-think are rare. Focus groups may be okay for driving small incremental changes, but innovation happens in leaps. I have seen many great ideas and innovations killed in a focus group.”

“When prospective clients don’t approach user experience with the preconceived notion of doing focus groups, this indicates a maturity of understanding about the role and the value of UX research in the marketplace.”—Carol Barnum

“I don’t consider focus groups to be part of the UX toolkit,” adds Carol. “But many initial inquiries from prospective clients show that they are erroneously thinking that they need a focus group to understand their product’s user experience. Whenever clients pose the question to me, ‘Do you do focus groups?’ I respond by asking what it is that they want to understand about their users. That always leads to a discussion of the difference between focus-group studies and usability studies. Then, the next question invariably is: ‘So, how many people do you schedule for a session?’ That question invariably leads to a deeper discussion about the value of usability testing in uncovering the experiences of a single, targeted user during an hour-long session; then seeing how the experience works for a second user, a third user, and so on. Finally, that conversation invariably leads to the question, ‘But how can you get valid results from five or six—or eight or ten—participants?’ which invariably leads to the research by Nielsen and others on why five is enough.

“When prospective clients don’t approach user experience with the preconceived notion of doing focus groups, this indicates a maturity of understanding about the role and the value of UX research in the marketplace. To this day, the large number of prospective clients with whom I talk who have never done any user research other than market research continues to indicate that we are still a young discipline. Nevertheless, the fact that they want to do user research is an encouraging sign that more and more are understanding that they need to know about their users and understand user experience. Since I always anticipate the question about focus groups, I’m prepared to educate clients on the differences between market research and user research. But I will celebrate the day that this question goes away.”

“One UX practice that I wish would disappear is product design that is based on specific feature requests from users and clients.”—Drew Davidson

“One UX practice that I wish would disappear is product design that is based on specific feature requests from users and clients,” replies Drew. “One of the most common expectations that prospective clients have is that their UX design agency will ask customers and internal stakeholders what they want and will then build out those feature requests. Many times, clients even have a feature list that they’ve gathered before beginning a project, and they expect the firm to design from that alone.

“Unfortunately, many UX firms will actually go ahead and do just that, so they fail to discover and solve the root problems that a new or evolving product should address. At best, that approach forces additional, incremental improvements; at worst, it causes feature bloat. ÄKTA’s approach is to observe, discover, and understand user behavior rather than simply ask users what they want. By doing this, we’re really able to identify the core challenges that users will encounter with a product—either now or in the future—and that knowledge allows us to architect and design solutions that overcome those challenges. We strongly believe that this approach leads to better, more efficient products that fulfill customer needs at a level much deeper than one can articulate in a quick feature request survey.”

Return on Investment (ROI)

“I wish that people would stop asking about the return on investment for user experience or usability.”—Jim Ross

“I wish that people would stop asking about the return on investment for user experience or usability,” answers Jim. “While, over the last several years, it seemed that requests for us to prove ROI had gone away, lately I’ve seen this question returning with some clients. In the early 2000s, when I was working on internal applications for a large corporation, we spent a lot of time trying to convince people of the ROI for usability. In theory, it was easy to demonstrate that an organization saves money if usability improvements save seconds or minutes of employees’ time, and you multiply that time by the number of employees using an application and the value of their time. But it was nearly impossible to get the actual figures because we never had accurate before and after measurements of performance. The irony was that it would have required a great deal of extra work and money to gather the metrics to show that usability improvements saved money. And no matter what we came up with, a skeptic could easily poke holes in it.

“I used to wonder, ‘Couldn’t people just stop asking for ROI and accept the common-sense notion that improving usability is worthwhile?’ Well, eventually they did. Usability and, later, user experience became widely known practices that are respected and valued in the business world. It wasn’t necessary to prove ROI anymore. People just accepted that a good user experience was valuable and essential. But lately, I’ve seen this concern creep back into our larger projects, so I wish that ROI would just go away.”

Roles and Politics

“ For those of us captured within organizations, opportunities to take the lead are growing every day. It’s time to step up to these leadership positions and take responsibility for all that such a shift implies….”—Leo Frishberg

“Rather than look at any specific technique, tool, or vocabulary,” responds Leo, “I would prefer to look at what we could start, continue, and stop in terms of behaviors and politics. For those of us captured within organizations, opportunities to take the lead are growing every day. It’s time to step up to these leadership positions and take responsibility for all that such a shift implies: confidence, forward thinking, inclusiveness, and responding to the push-back that such leadership always invites. We need to continue to model a way of thinking that is often rare in engineering-driven—and even marketing-driven—organizations, using lateral, synthetic, and abductive reasoning. We need to stop playing the victim card—how the world doesn’t listen to us, marginalizes us, and fails to appreciate the value that we can bring.”

“I am an opponent of delegating user experience to one person or role.”—Jordan Julien

“I am an opponent of delegating user experience to one person or role,” responds Jordan. “I believe that each team member should bring his or her own flavor of user experience to the table—and organizations should encourage them to do so. In some recent meetings, I’ve been introduced as ‘the UX’ on the project. I’m the user experience? What a big responsibility. We need to stop obfuscating our roles, start evangelizing user-centric design, apply HCI principles throughout our projects, and stop trying to own user experience.”

“Schedule overruns, budget overruns, and unmarketable products … stem from this failed ‘lean’ approach to product development.”—Shane McWhorter

“I wish the use of the term user experience as a catch-all for all of the skills that are required for product design and development would go away,” says Shane. “Organizations wanting to staff a product design and development team with a single UX specialist to cut product development costs are exploiting this catch-all term. Formerly, a product design and development team might have included product managers, business analysts, information architects, usability analysts, usability testing specialists, user interface developers, creative directors, and graphic artists—as well as brand managers and many other marketing specialists. Each of these is a specialized role—in many cases, having its own four-year degree.

“Now, more often than not, I see openings at organizations for a single UX Architect, requiring all of these skills and many years of experience in each of them—though perhaps requiring only a four-year computer science degree—but priced as a mid-career graphic designer. These are the organizations that publicly lament the shortage of ‘qualified candidates’ in the marketplace due to their failure to find a ‘purple squirrel.’ In my experience, this trend is the single most detrimental side effect of using the term user experience, because of the resulting schedule overruns, budget overruns, and unmarketable products that stem from this failed ‘lean’ approach to product development.”

Terminology and Semantics

“The UX practice that I think should go away is a consequence of two interrelated conditions—our insatiable love of UX jargon and our zeal to overpopulate an already littered field with redundant terminology.”—Ronnie Battista

“The UX practice that I think should go away is a consequence of two interrelated conditions—our insatiable love of UX jargon and our zeal to overpopulate an already littered field with redundant terminology,” replies Ronnie. “We are not doing ourselves any favors by talking in terms that our companies and clients struggle to understand.

“We—and I very much include myself here—suffer from a disease I call Heuristica Jargonistica—the seeming inability of a UX person to get out of his or her our own way in communicating our value and what we actually do. Having suffered from this affliction for years, I do think that I’m finally starting to get more control of this audible unpleasantness. For example, I’ll use the term heuristic these days only when someone else uses it—typically either a self-congratulatory ‘I read a UX book!’ jargon-dropper or someone who was a bit bewildered when the last UX person tried to explain or sell a heuristic evaluation—which of course, in most cases, was actually an expert review—another favorite. I do think lean UX—which is RITE (Rapid Iterative Testing and Evaluation), which is discount usability—does a much better job of humanizing language. I prefer ‘get out of the building and talk to people or watch what they do’ to ethnographic research or contextual inquiry. I don’t think I’ve mentioned bipolar emotional response testing in at least four years.

“Consider journey mapping—well, okay, not journey mapping, experience mapping—which is really about holistic—or is it omnichannel?—customer experience. But really, it’s for customers, so it’s customer experience mapping, or wait, no, it’s a customer journey map. So, hold on, come to think of it, it’s more like customer experience journey mapping. And we’re designing this, right? For employees, too? So, okay, what we’re doing in that case is employee experience journey mapping design. Isn’t this really service design though? Experience design?

“As a serial neologist and lover of acronyms, this is like crack. Or perhaps crack should be replaced with lean. Not the lean that seems to be in any business process that people want to simplify—for example, UX, manufacturing, learning, startups—I mean the drug made of cough syrup, soda, and jolly ranchers. I actually prefer its other name, Purple Drank, as it’s apparently purple and you can drink it.

“Companies need our skills to help them reach their goals—regardless of what we call our profession. Getting stuck in semantics helps nobody.”—Janet Six

“This is getting far too complicated. Perhaps we should start over. With journey mapping in mind, I hereby suggest a new name—the acronym for which helps explain, in human terms, what we really know about such naming conventions: Journey Alignment Customer Kinetics of Strategic Holistic Interactive Touchpoints, or JACKSHIT.”

This conversation reminds me of a frequent discussion on the IxDA discussion board in the early days of the organization: “What should we call our discipline?” Discussion was heated and—notably—the rate at which new members joined the group slowed down during these times. Companies need our skills to help them reach their goals—regardless of what we call our profession. Getting stuck in semantics helps nobody.

Lorem Ipsum—Huh?

“I hate lorem ipsum text and placeholder images.”—Caroline Jarrett

“This is an easy one for me,” replies Caroline. “I hate lorem ipsum text and placeholder images. Participants in usability studies don’t understand them, which undermines the value of early testing. And they distort the design.”

Fewer Clicks

“I wish the idea that fewer clicks equals a better experience would just go away. This is another example of people trying to dictate a solution before properly understanding the problem.”—Jordan Julien

“I wish the idea that fewer clicks equals a better experience would just go away,” says Jordan. “This is another example of people trying to dictate a solution before properly understanding the problem. We should understand that optimizing user workflows involves several things, including how difficult it is for the user to click. Consider whether it would be better for a user to be able to make five clicks without thinking or make just one click after expending copious amounts of mental energy skimming the page and mulling the available options. Typically, users prefer to make quick, satisfactory decisions rather than mull over the consequences of each decision.”

This problem is reminiscent of a phenomenon in Graph Drawing (GD) Visualization—which had grown from the field of VLSI circuit design where:

  1. It was necessary to draw a circuit as small as possible for manufacturing reasons.
  2. Users typically did not need to view the circuit in use.

Thus, many GD algorithms focused on producing visualizations that were optimized for small size. However, in practical applications, having a bit of extra space actually improved visualizations. So, be sure that your UX methods serve their true purpose. Do not optimize methods in a particular way just because something has always been done a certain way.

Reference

Bowles, Cennydd, and James Box. Undercover User Experience Design: Learn How to Do Great UX Work with Tiny Budgets, No Time, and Limited Support. Berkeley: New Riders, 2011.

15 Comments

Great article. Focus groups, fewer clicks, and design by feature request resonate a lot with me. Somehow, those things resonate with managers. Another problem I’m seeing today is a preponderous of jobs for a UX Designer that describe what used to be called a Web/graphic designer. The other aspects of UX seem to be given lip service. I would think any position with UX in the title should require a good amount of research and analysis.

I can’t say I agree with Jessica’s sentiments regarding lean UX. When I’ve seen Jeff talk about “getting out of the building,” he’s always been talking about going to real/potential end users, not just the hobo on the corner who will spend 20 minutes with you for a ham sandwich. So the miner example just doesn’t fit. He also poses the users themselves as an assumption that needs validation, where traditional, controlled usability testing typically involves very detailed selection that can potentially lead you to testing with users who are never going to really dive into your product.

You can also test in a lean manner though a controlled lab environment. Lean UX isn’t about dictating testing methods; it’s about gaining insights more frequently and course correcting.

“Imitating Other Companies.” I also have a distaste for this view of the world. I can’t tell you how many times I’ve been told by a client, “Let’s not reinvent the wheel. Those other sites are working fine, and all my competitors have the same layout.” What they are really asking for is that you lock down any innovation whatsoever and clone the competitor’s environment. There is a possibility that some innovation will give a client the advantage over the herd of sheep. Still it’s always an uphill task to convince people of the opportunity.

One thing I wish would go away is the politics surrounding UX and the battle for power between designers and developers. When a client recently said to me, “You don’t need to understand, just do as I say!” I realized that UX still has a long way to go in some places. The questions that come to mind are: Do I want to spend my efforts on these politics? Should I try to educate unwilling clients? Should this really be part of my job?

Jake, I absolutely agree with you regarding lean UX, and Jeff and Josh’s intentions regarding research.

My concern is not with lean UX as the originators pitch it, but rather the broader “lean UX practice” that is likely to evolve. I worry that too many practitioners—and clients?—hear “lean UX”—without understanding the origins—and see it as an excuse to perform less rigorous research. But I guess in that way it’s like any new technique or tool: it can be used or abused. Let’s hope it’s more of the former than the latter.

Thanks for taking the time to comment.

Great article and so thought-provoking. How wonderful to have all of these perspectives from experienced professionals.

Re: Ziv Lazars questions about a client saying, “You don’t need to understand, just do as I say!”—

It depends. ;) Do you have the patience and stamina to fight through the politics? Do you enjoy the challenge of gaining influence with hard-nosed, demanding people? Are you excited about educating clients who don’t understand—and maybe don’t care to understand—UX? If those challenges are appealing to you, you can throw a rock in any direction and find work as a UX designer. I think yes, most folks would expect this to be part of your job.

If you lack the patience or find the struggle to gain influence tiresome, you may be able to avoid it by working with like-minded people. (Say, as an in-house designer in a boutique, design-focused agency.) But you will be severely limiting your career options by attempting to avoid any kind of power struggle. Frankly, there are a lot of fearful and ignorant people out there hiring UX designers. Many of them work in organizations with a lot of money.

I’ll suggest that perhaps it should be your goal as a professional to build yourself up to the point where if any client ever says “do as I say” to you, you feel comfortable firing them.

Sustainable, practical working relationships between designers and clients involve partnership, collaboration, and trust. Earlier in my career, I was often in situations where I didn’t feel any of that from a client / manager / director / peer. They simply wanted me to do their bidding without question. I didn’t know how to respond, and I was frustrated. But as I have learned how to navigate politically treacherous waters and have become more skilled and confident in my craft, I now find this kind of communication unacceptable. If a freelance / consulting client says this to me, I don’t put up with it. I talk with them about the nature of our collaboration. If they persist in attempting to dictate too much to me, I calmly fire them. (Yes, you can fire your clients. You have that much power.) Thankfully, it doesn’t happen often. But I always make sure my contracts have an early termination clause that protects me!

If this happens where I am employed at a Fortune 500 company, I might work harder to build friendships with power brokers in the organization to gain influence. If the “do as I say” attitude continues, I might choose to walk away and find work elsewhere. Especially in this environment where there are so many good opportunities for UX designers. Nowadays, if you can spell UX, you have employment options. At least it seems so in my neck of the woods. Use that to your advantage.

All of which brings me to a thing I’d like to see go away: the lopsided or one-sided view of the UX designer / client relationship. Not just from the client side—but in the minds of designers! The client or internal business stakeholder does not hold all of the power. The very fact that you, the designer, are there indicates that they need you. Don’t hesitate to leverage your position. Learn how to politely, professionally, demand respect. And be worthy of the respect you demand.

This is hands down the best article I’ve seen on UX in ages. The profession needs to take a look at itself and see how ridiculous it can be sometimes.

I’m an aspiring PR professional, but lately I have been getting into UX and thinking maybe that is a calling for me. But I have no idea where to start. Obviously some design classes, but how do you really make it in this obscure field? Any advice would be much appreciated! Oh and my twitter handle is @rawnstet

Great article; one of the best! Thumbs up on Focus Groups, Semantics and Terminology, ROI…okay, all of it. Here’s a new term: Technical User Requirements Document, or TURD. Thanks again for such an engaging article!

Thank you for your great feedback and discussion! I am so glad that you are enjoying this column. More questions have come up in the comments. Let the discussion continue! :-)

Excellent article; best one I’ve read on UX in quite some time. It rang true for me on many levels. Gets double points for not being a list—“10 things you can do to improve your UX!” Gah.

I am not a salesperson or a persuader, so navigating politics is always stressful for me, and I truly wish it wasn’t a part of my job. Defining what UX is and isn’t—also really wish this wasn’t a part of my job. Being pressed for wireframes when we haven’t even decided the IA of the site yet. Focus groups = usability. Too many acronyms. I’m on board with it all!

Something I’d like to see go away is the notion that 5-9 participants in a usability study uncover 80% of the problems. If this were actually true, then all products that have undergone usability testing would have no support issues, incredible clickthrough (100%), and never need to be fixed again.

The Nielsen research shows only that, with 5-9 experts doing a heuristic evaluation, following an agreed upon and standardized process, they tend to uncover only a certain number of problems before they start to repeat each other.

Further research that actually looks at usability studies—and not heuristic evaluations—shows that for every new study, set of tasks, or researcher, completely new problems are found.

See the research by Rolf Molich—which he has code-named CUE—for comparative usability evaluation—and work by Jared Spool on his Web site Usability Engineering (UIE.com).

The CUE studies show that, when 7 or more research teams evaluate a product or set of features—or even examine video recordings of a study—you get 7 sets of non-overlapping results. Not only is this not 80%, the amount of overlap rarely reaches 10%.

So why do we do usability testing? Like design critiques—we do it to get insights we cannot get any other way. To learn about gaps in our thinking, mental models, user types, workflows, and so on.

It is the fastest and most economical way to uncover the most information about how our products might be used and can be improved.

A big caveat in doing usability testing is to include people who really, actually need the product for their tasks.

I watched videotapes of a usability study of do-it-yourself moving services—like U-Haul—in which the first participant said, “I’d never move myself. I’d always hire movers to pack and transport….”

That’s the point to make up some excuse like the camera is broken, and excuse them from the study. Let them go, destroy their responses, and have a long and serious discussion with the participant recruiter.

The only participants for a phone company are ones just about to buy or change plans; for an ecommerce site, people ready to make a purchase of that type of product, and so on.

“Imitating Other Companies” and “Fewer Clicks”—totally agree! Excellent!

Wonderful article with great opinions. Loved the focus group and lean UX comments. I think Carol was showing that a frank discussion about focus groups versus usability testing might clue clients into why focus groups can provide inadequate results. I’m imagining how that discussion might go, but if anyone has additional strategies for convincing clients that focus groups are not the way to go, I’d love to hear it.

Great discussion! Let’s keep going!!

Join the Discussion

Asterisks (*) indicate required information.