The following experts have contributed answers to this question:
- Mike Hughes—User Assistance Architect at IBM Internet Security Systems and UXmatters columnist
- Whitney Quesenbery—Principal Consultant at Whitney Interactive Design; Past-President, Usability Professionals’ Association (UPA); Fellow, Society for Technical Communications (STC); and UXmatters columnist
- Mary Theofanos—Chair of the Industry Usability Reporting Project Working Group at the National Institute of Standards and Technology (NIST)
Q: My task at hand is to write nonfunctional requirements for the usability quality of a large, new Danish system for handling social services in the public sector. There is an existing product, but it is 25 years old and runs on a CICS mainframe. The classic answer to my question is to decide upon usability factors such as learnability, understandability, or efficiency, then establish metrics like the following:
- Six out of ten novice users shall perform task X in Y minutes.
- At most, one in five novices shall encounter critical problems during tasks Q and R.
But still, where do I start? And how do I choose the numbers in these metrics—for example, six out of ten and one in five? Some literature suggests requirements should be written more as general concepts or ideas for interactions, but I need these usability requirements to be strong, because we’re outsourcing the design. Therefore, we must use these requirements to assess and secure the ongoing quality of what the vendor delivers.
So far, it seems to be very difficult for my organization, with no direct prior experience, to decide what metrics or requirements to use. I will suggest something, but I must be prepared to discuss, fight for, and change what I’ve proposed. So, what experience can I draw upon? Can you offer any advice for this process? Where do I start?—Ole Gregersen, Information Architect
Whitney responds, “First, I want to applaud you and your organization for trying to create a way to ensure that the system will meet specific user experience goals. This is especially important when you are outsourcing the design work and can have little control over the process except by setting requirements for the outcome.”
Mary shared some work they’re doing at NIST: “The Visualization and Usability Group (VUG) at NIST has also recognized the need for defining usability requirements in enough detail to influence design and allow us to validate them. Working with partners in industry and academia, we have been working to develop guidelines for specifying usability requirements. The working group developed the Common Industry Specification for Usability Requirements (CISUR).”
The Difficulty of Writing Usability Requirements
“Your classic examples are just examples, so I won’t pick on them too much,” says Whitney. “But you are right that they are not very helpful. For one thing, they raise more questions than they answer—for example, What is a novice user? What is a critical problem?”
Mike offers this opinion: “I think we overemphasize metrics when it comes to usability. They often introduce problems in terms of the validity and reliability of the data. First, ask whether a metric is a valid measure of usability. For example, the amount of time it takes to perform a task is often a usability goal—as in your example Six out of ten novice users shall perform task X in Y minutes. What if all of the participants took more than Y minutes to complete the task, but everyone said, ‘This is the best application for doing this task I have ever experienced. I’m recommending this product to all my friends.’ Does the product fail to meet the requirements regarding that task? What if all participants perform the task in less than Y minutes, but everyone says, ‘That was a horrible experience. I would never buy this product.’ Does the product satisfy requirements?
“The problem with these metrics is exactly the one you have articulated: What is a good number? Such measures are invalid unless
- you can derive a metric from prior data—For example, you might have Web metrics that show, if users cannot complete a purchase in Y minutes, the abandonment rate goes up.
- a metric is intrinsic to the product—For example, if a medical team cannot complete a triage procedure in less than Y minutes, the patient dies.”