Much of what appears to be senior management’s irrational behavior in regard to user experience in general and usability testing in particular results from their inability to get their head around how to measure the value of user experience or usability testing. In many companies, there is now strong demand to improve product usability, but most executives lack sufficient understanding of how to measure the effectiveness of UX efforts. As Peter Drucker has said, “If you can’t measure something, you can’t manage it.” This inability to measure UX efforts tends to lead to inaction—or worse yet, micromanagement by people who think they are Steve Jobs, but lack his UX savvy.
Failing to Defend Small Sample Sizes
I believe one key problem is that, as UX professionals, we naturally strive to come up with simple descriptions of complex things, but often fail to do so. We need to keep in mind that it is important to avoid oversimplifying to the point where we confuse both ourselves, and those we collaborate with. This is especially true now that the Internet lets us broadcast our thoughts to a broad audience, in a format where casual readers Googling for quick answers often consume them without much reflection.
Let me give you an example. Last year, a contact in India asked me to review a presentation that cited Steve Mulder’s book The User Is Always Right: A Practical Guide to Creating and Using Personas for the Web. Steve’s book contains a chart that categorizes usability testing as a qualitative research method. My contact was using that chart to explain user research to the next generation of UX professionals there. Here’s the problem. That chart is very misleading; in fact, I’d say it’s just wrong.
My contact in India sent an email message off to Steve, including my comments and CC’ing me. Steve admitted he’d oversimplified things, because in his experience, the use of the term quantitative research confuses many teams when researchers apply it to small-sample studies and that wasn’t really the topic of his book. He also noted that, on the teams with which he’s worked, most conduct usability testing “more as interviews than observational studies, unfortunately.”
My response to Steve was that he should send his confused coworkers over to Jeff Sauro’s site instead of glossing over the issue, because Jeff does an excellent job of explaining small-sample statistics for use in design on his blog Measuring Usability.
I copied Jeff on my reply, as well as some other friends on the UPA Board of Directors, who have been discussing training and documenting best practices a fair amount recently. I applaud Jeff for writing several good blog posts on the topic shortly thereafter, including “Why You Only Need to Test with Five Users (Explained)” and “A Brief History of the Magic Number 5 in Usability Testing.”
What’s the Impact of A/B Testing’s Popularity on User Research?
So, how does all of this relate to A/B testing? Here’s how. During the recent economic downturn, several of the folks I’ve worked with over the years who are excellent user researchers have found themselves out of work. Why? Well, I suspect the popularization of A/B and multivariate testing could explain some part of this. The perception exists in the minds of many executives that A/B and multivariate testing provide better data—or more specifically, data that is quantitative—and thus, eliminate the need to do any other type of user research. This perception concerns me, as it should any UX professional.
Interestingly, unlike traditional usability testing, A/B and multivariate testing are now familiar to many of the executives I talk with today. I believe that’s because the people who have gotten involved with Web analytics tend to work in marketing research and, not surprisingly, that means they’re pretty good at communicating the value of a service like A/B testing. Or, at least, better at marketing the value of A/B testing than the human-factors types who tend to be experts in small-sample usability testing are at communicating its value. The result? Many executives—and even some UX teams—have latched onto A/B testing as if it were some sort of silver bullet. It’s not. However, as all UX professionals know, perception is often more important than reality, especially when people in powerful positions—many of whom are statistically impaired—hold a particular perception.
Instead of reiterating the points Jakob Nielson made back in 2005, in his article on the pros and cons of A/B testing, “Putting A/B Testing in Its Place,” let me add a few points I don’t think anyone has communicated well so far.
