UX Analytics, Part I: A Call to Action

By Kristi Olson

Published: March 21, 2011

UX analytics—a marriage of right-brain, deep-dive, empathic UX research and left-brain, quantitative analysis of users’ actual behaviors”

As UX researchers, our goal is to identify customer pain points and obstacles in a given workflow or process, then tell a compelling story about their risks and provide general recommendations for alleviating those risks. But after eight years in UX research—even having mastered the arts of compelling storytelling and building stakeholder empathy—I still never quite saw the results I’d hoped for. Worse, my work required me to move quickly onto the next project, never to hear again about the outcomes of projects past.

However, my experience on one project was enlightening: When Web analytics data disproved my post-usability test hypothesis, I realized the importance of my being data driven in my practice—as opposed to my just providing UX research data. This revelation was the starting point of my reinvented career in UX analytics—a marriage of right-brain, deep-dive, empathic UX research and left-brain, quantitative analysis of users’ actual behaviors.

This transformation has provided a new depth of power to my storytelling, with the result that some of my key issues and recommendations have moved rapidly up to the top of our development team’s list of priorities and are now considered key business initiatives. For example, instead of my saying, “Users are confused about creating an account,” I’d instead say, “Users are confused about creating an account, and 36% of them are abandoning the process, causing an estimated revenue loss of $650K annually.”

In this article, which is the first in my series on UX analytics, I’ll define and describe the value of UX analytics and initiate you in this very fruitful approach to UX research.

The Business Impact of UX Analytics

“At the heart of UX analytics is business impact. … Our recommendations for usability improvements must compete with other projects that have revenue projections tied to them.”

At the heart of UX analytics is business impact. My stakeholders have rarely denied the importance or even criticality of the obstacles and inconveniences that our Web sites present to our customers. But, as UX professionals, our recommendations for usability improvements must compete with other projects that have revenue projections tied to them. My call to the UX community is that we must compete on the same playing field with other projects by aligning our arguments with the same business logic.

The consequence of my doing this level of business analysis is that I am now more deeply involved in projects, I am often consulted by stakeholders, and my participation on a project is integral to the implementation of my recommendations. In addition, I am responsible for the post-implementation measurements, which enables me to recommend further design iterations if both the data and my design expertise indicate there’s still opportunity to improve the user experience.

This approach has afforded me a better platform for advancing awareness of user needs—whether through further usability testing, customer interviews, or redesign efforts that employ iterative prototyping and testing methods. Now, my case might be, “Users are confused about creating an account, and 36% of them are abandoning the process, causing an estimated revenue loss of $650K annually, so I recommend that we do a deep dive, through a series of usability tests, to understand why users are abandoning.”

Another powerful benefit of UX analysis is that my clients are now key advocates for the changes that I’m advocating. They’ve become empowered, engaged negotiators with development teams, persuading them to implement the solutions that best balance user and business needs.

What Is a UX Analyst?

“Marry multiple sources of data … with user-centered design best practices to deliver actionable insights and recommendations.”

To give you an idea of what my objectives are as a UX Analyst, here are a few bullet points from my newly crafted job description:

  • Marry multiple sources of data—such as customer experience management, Web analytics, customer satisfaction, call center feedback, and primary or secondary research—with user-centered design best practices to deliver actionable insights and recommendations.
  • Do strategic planning for data gathering and measurement that support Ebusiness initiatives and proactively identify needs for measuring user experiences.
  • Clearly communicate the business and customer experience opportunities, and deliver measurable recommendations that increase value to the business and the customer.
  • Lead the design of new and evolving UX and Voice of the Customer (VOC) dashboards.
  • Draft UX documentation such as wireframes, flowcharts, and taxonomies when useful as a way of supporting recommendations.

Today, I report to our Digital Analytics Director, who was formerly known as our Web Analytics Director and changed the team’s name to communicate that it encompasses both quantitative and qualitative customer behavior research and analysis. My key clients are the Site Managers, each of whom owns of a particular ecommerce Web site and is responsible for its development roadmap, site revenue, and key performance indicators (KPIs). All of these teams reside in Ebusiness.

Getting Started with UX Analytics

“Build a trusted partnership with someone on your company’s Web Analytics team….”

Here are some immediate steps you can take to get started down this UX analytics path. I suggest that you approach UX analytics by taking these steps in the order I’ve listed them:

  1. Build a trusted partnership with someone on your company’s Web Analytics team who would be willing to meet with you for an hour on occasion, to pull some reports and explain them to you.
  2. Find out which Web analytics tools your company uses, do a little research on them to understand their utility, and start becoming familiar with their use.
  3. Take an Excel class or some tutorials to become comfortable with the mechanics of manipulating data—like calculating averages, sums, and divisions of data. (You don’t need to worry about formulas and pivot tables right now. Macros are not something I work with.)
  4. Build a strong partnership with your User Acceptance Testing (UAT) analysts who are responsible for the Web sites of interest. They have direct insights into bugs and system issues that you should become intimately aware of. Sometimes recommendations may include fixing a bug that has been sitting at the bottom of a queue for a while.
  5. Reach out to your VOC/customer satisfaction survey analysts to get on their distribution list, including getting open-ended responses from customers.
  6. Reach out to your call center team to get on their distribution list for reports indicating Web site problems, including getting customers’ open-ended responses from call center representatives.
  7. Steep yourself in these open-ended responses. I suggest spending no more than 30 minutes at a sitting. The goal is not to read all of those responses, but to make a practice of rapidly auditing them to build your tacit knowledge of your customers’ needs and experiences. Ultimately, doing this could build your position as one of the authorities on your company’s customer needs.
  8. Consider other sources of data that could reveal user insights and meet the analysts or get on their distribution lists, if they have any. Other sources could be optimization testing or social media analysis.
  9. Brush up on various methods of online qualitative user research and the remote tools the entire UX community is discussing today, including heatmaps, eyetracking, clicktracking, unmoderated usability testing, and remote usability testing. For a fairly complete list of tools, check out the article “24 Usability Testing Tools,” by W. Craig Tomlin.

In Conclusion

“I’ve experienced the power of advancing UX findings and recommendations by tying them to business impact.”

I’ve experienced the power of advancing UX findings and recommendations by tying them to business impact. So can you. Just building a trusted partnership with someone on your Web Analytics team may be empowering in itself and open some new doors, spurring thought and innovative ideas for change.

In Part II in this series, I’ll tell you how to get started with business impact analysis by creating a business process funnel.

3 Comments

Wonderful article, Kristi.

I really believe that UX, as an industry, needs to move more into this direction. In my 15 years experience, I’ve always found the same problem/challenge: How to make UX accountable. Let me share a recent anecdote:

I was talking to a highly experienced UX Analyst from one of the world’s leading technology research and business insights companies last week. He told me that, according to his personal experience, most mid-to-large enterprises still fail pretty badly when it comes to managing UX. He said that C-level execs at many of these companies don’t even know they have a UX issue or how to solve it. He also said that many of them know they have a problem, but don’t do much about it because… “no one gets fired at these companies for having a bad UX.” Pretty amazing, huh? He went on saying that ‘this will be changing over time, thanks to cases like Nokia or Sony getting crashed by Apple.’

Meanwhile, if we UX pros are to be respected like other pros—Marketing, for example—in my opinion we need to make UX accountable. That’s one of our biggest challenges.

Thanks again for your article!

Alfonso de la Nuez

Co-Founder at UserZoom

Alfonso—You make an excellent point. Per the example I mention in the article, where Web analytics data countered one of my usability hypotheses, I realized that I could provide better decision support and richer analysis by expanding my datasets to other sources I hadn’t previously considered. This new level of curiosity and investigation has made me become more accountable for my recommendations and our team’s designs. Once my recommendations are put into action, I now have a process to not only measure the business’s progress, but my own progress.

Awesome article, Kristi!

I’ve long been an advocate of looking outside your cube to find ways to be more effective in UX. Silos are what create many UX problems, but it seems in UX we often point fingers rather than collaborate.

Liam Friedland and I talk about this in the tutorials and workshops we’ve given on UX strategy. Check our presentation out if you are interested. We started doing these almost 10 years ago when strategy and management were never discussed at UX conferences.

The key to an effective UX strategy is realizing that product development is a team sport, and UX is no exception. That includes considering all parts of the organization as partners and talking about what matters to executives—why UX matters in terms of money.

Jon

UX Innovation LLC

Join the Discussion

Asterisks (*) indicate required information.