Aligning UX Issues’ Levels of Severity with Business Objectives

By Paul J. Sherman

Published: October 4, 2010

High, medium, and low don’t begin to sufficiently explain the potential brand and business impacts usability issues can have.”

Many of us in the field people now generally refer to as user experience have long used levels of severity as a means of indicating the criticality of a product’s or service’s usability issues to clients.

Over the past several years, I’ve grown increasingly dissatisfied with the vague and somewhat solipsistic nature of the gradations UX professionals typically use to describe the severity of usability issues. High, medium, and low don’t begin to sufficiently explain the potential brand and business impacts usability issues can have.

After incrementally iterating on several existing classifications of severity, I finally decided in late 2008 to simply create some new ones, which I’ll present in this column. For lack of a better term, I call them business-aligned usability ratings.

What’s the advantage of my ratings? In short, business-aligned usability ratings help me communicate the actual business impacts of the issues we identify during usability testing. They include information about how a user experience issue can affect an organization’s brand equity and revenue.

Before I present these ratings, let’s first review some definitions of usability and user experience, as well as some common classifications of usability or user experience issues.

Defining Usability—Again

“Business-aligned usability ratings help me communicate the actual business impacts of the issues we identify during usability testing.”

Let’s go to the more authoritative sources first. The ISO standard for usability (ISO 9241-11) defines usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.”

Another popular definition is Jakob Nielsen’s conceptualization of usability as a “quality metric” with five components:

  • learnability—How easy is it for users to accomplish basic tasks the first time they encounter a product or service?
  • efficiency—Once users have learned a product or service, how quickly can they perform tasks using it?
  • memorability—When users return to a product or service after a period of not using it, how easily can they re-establish proficiency?
  • errors—How many errors do users make, how severe are those errors, and how easily can they recover from the errors?
  • satisfaction—How pleasant is it to use a product or service?

In my own practice, I’ve typically defined usability in a broad context. Here’s my working definition:

Your intended users can accomplish what they’re trying to do with your product or service.

However, as you’ll soon see, I’ve oriented my ratings of usability issues with a tighter focus on business problems and their solutions.

Defining User Experience

“I’ve always liked Morville’s definition of user experience best, but found even it somewhat lacking because—like other definitions I’ve run across—it didn’t seem to put enough focus on the business outcomes.”

ISO 9241-210 defines user experience as “a person’s perceptions and responses that result from the use or anticipated use of a product, system, or service.” Other definitions follow this perception/emotion-centric view, as we can see in Wikipedia’s definition: “User experience is about how a person feels about using a system.” Others adhere to a component-based definition of user experience—for example, Peter Morville’s user experience honeycomb, which comprises usefulness, usability, desirability, value, accessibility, credibility, and findability.

I’ve always liked Morville’s definition of user experience best, but found even it somewhat lacking because—like other definitions I’ve run across—it didn’t seem to put enough focus on the business outcomes. So rather than simply creating and advocating for yet another definition of user experience, I began evaluating usability or user experience issue ratings with a focus on an organization’s business objectives.

Benefits of Business-Aligned Usability Ratings

“Business-aligned usability ratings … help UX professionals focus on real issues—that is, issues whose resolution would have an actual impact on the user experience rather than issues that may be more academic.”

Business-aligned usability ratings offer UX professionals several advantages, as follows:

  • They connect the dots between a given usability or user experience issue and the business ramifications of that issue, helping both you—as a UX professional—and your stakeholders understand what is at stake if you leave an issue unaddressed.
  • They help UX professionals focus on real issues—that is, issues whose resolution would have an actual impact on the user experience rather than issues that may be more academic.
  • They encompass both brand equity and revenue impacts, which other ratings don’t seem to cover adequately.
  • They are behaviorally anchored—that is, they are based on observable, well-defined, or inferable criteria.

Rating Usability or UX Issues

Here are my working definitions for four levels of severity for usability or user experience issues.

  • critical severity—An issue of critical severity would definitely result in users’ being unable to complete their intended task, as well as an immediate, noticeable, and significant negative impact on an organization’s brand equity, revenue, or profitability.
  • high severity—A high-severity usability issue is one that would likely result in users’ not being able to complete their intended task and, from a business perspective, negatively affect an organization’s brand, revenue, or profitability.
  • medium severity—A medium-severity usability issue would be likely to significantly impede or frustrate users, but probably would not prevent users from eventually accomplishing a task. It might also negatively affect an organization’s brand, revenue, or profitability.
  • low severity—A low-severity usability issue would likely present some type of momentary or transient difficulty or result in confusion for users, but would not prevent users from accomplishing their task. It would have no discernible effect on an organization’s brand, revenue, or profitability.

Why Consider Both User Experience and Business Outcomes?

“To remain competitive, organizations must provide excellent, not merely passable user experiences.”

In a word: alignment. The competitive environment for digital products and services has never been more tenuous for market incumbents or more full of opportunity for market entrants. To remain competitive, organizations must provide excellent, not merely passable user experiences. Loosely paraphrasing Jakob Nielsen, it is wise to remember two maxims:

  • All things being equal, most users spend most of their time on other sites—with the exception sites like Facebook, I’m guessing.
  • If a user isn’t happy with how your site supports their goals, they will leave for greener pastures.

All of this is just a fancy way of saying that user experience is an important competitive differentiator. To adequately leverage user experience design and evaluation, we need tools that enable us to better measure user experience. The rating scheme for usability or user experience issues I’ve presented here is one such tool that can help you to communicate the real-world, financial consequences of such issues to business stakeholders more effectively—as well as the potential costs of not addressing them.

3 Comments

Good article on an important topic, Paul. My only beef with your ratings scale is that they line up severity of business impact with usability impact—ability to accomplish goal and impact on brand, and so on. Which then begs the question: why not just go with High, Medium, Low, and Trivial?

I had thought the point you started out to make is that these two important issues are not always in alignment—that is, you might have a trivial usability issue that impacts business or vice versa. Perhaps a grid would be a way to line those up.

I like these new definitions because they make such a tangible connection between user experience, potential product success, and the bottom line, and they’re written in plain English. Nice work. :) I’m going to pilot them in my next usability study.

Great article, Paul. Thanks for sharing your methods and insights! I’ve found that I’ve tended to move away from assigning severity ratings lately for many of the reasons you outline here—too academic, too UX geek speak for my audience, and so on. The definitions you’ve presented are definitely a move in the right direction in making these more relevant.

I’m curious how you delineate UX issues that have no real severity associated with them when you’re reporting—that is, do you tend to group things by issues (with severity) and observations (without)?

Join the Discussion

Asterisks (*) indicate required information.