Top

How to Calculate the ROI of UX Using Metrics

July 9, 2012

Even experienced UX professionals often feel that they are not being heard by their clients, managers, and developers. Why? Many such problems come from our desire to be valued for our knowledge and skills alone and to have our expertise respected without question. But this desire conflicts with the reality in which we find ourselves. To overcome this problem, we need to demonstrate that we bring measurable value to the products for which we design user interfaces.

Armed with your understanding of a business and a calculator, here are a few ways in which you can prove your value as a UX professional and get the resources you need—whether budget, UX team members, or more time.

Champion Advertisement
Continue Reading…

An All-Purpose Gauge

The best way to find a common language with business stakeholders is to communicate in numbers. The language of numbers is familiar to people who plan projects or are responsible for business operations. They let you compare either before-and-after results or your client’s results with their competitors’ performance. But the most important thing is being able to tie the results of your UX design changes to money a client earns or saves. This is the best way to improve your ability to communicate the value of user experience to your clients.

What kinds of numbers make useful metrics for user experience? There are three types:

  • money earned—for example, conversions or ARPU (average revenue per user)
  • money saved—for example, support costs or task performance efficiency.
  • non-monetary results—for example, user loyalty or recommendations to friends

Using these metrics makes your communications more objective. You’re discussing not artifacts like wireframes, but business goals and roadmaps for achieving them. For example: “To increase conversions, we should rework five key pages of the checkout process—creating wireframes of our design solution and conducting usability testing. Then, once the changes are live, we’ll track performance indicators to measure whether our work was successful.” You can avoid your client’s endlessly questioning Is this what we really need? and your facing a series of painful attempts to convince them that your design decisions would satisfy customer requirements.

But, even more important, by using metrics and tying them to business indicators, you can transform projects that would have died at the sales stage—with a prospective client’s saying, “We’ll think about it”—to sales successes. These metrics can help your clients to economically justify their engaging UX professionals on a project, because their return on the money they’ll pay for your work will be the additional earnings or savings they’ll achieve. What metrics should you calculate? Let’s discuss some specific metrics and how you can calculate them.

Metric #1: Conversion Rates

The Web became an effective sales channel some time ago, and for many companies, it’s now their main or even their only channel. Therefore, this channel has to provide a reasonable payoff, which requires understanding how a sale is made and how a company’s Web site can support the purchasing process. What does its sales funnel look like? What actions must a prospective customer take on what pages to complete a purchase?

Conversions might measure the number of sales on an ecommerce Web site in comparison to the number of visits, the number of product requests customers submit on a bank Web site, or the number of new registrations for a paid service.

When you optimize a sales funnel’s performance, the result is increased sales. And you can measure the value of a UX designer’s contribution in additional profits for a client. If you can show these kinds of numbers to your clients, there’s a good chance that you’ll be able to convince them of your value.

How to Calculate Conversion Rates

Let’s take a travel booking Web site as an example. The key metric here would be converting visitors to customers—that is, the number of site visitors who successfully finish the booking process. Here’s the formula:

number of people who complete the booking process / number of site visitors * 100% = conversion (%)

Step 1: Starting Conversion Rate

Let’s assume that some initial generative research, a slight redesign, and its implementation would cost $10,000. The return on a client’s UX investment should cover at least this cost. But the return should really be greater than that to consider a project successful. To achieve an optimal return on investment, it’s necessary to know how much profit each new order would contribute. Using hypothetical numbers, if each new order were $100, your client would need to bring in 100 new customers.

$10,000 / $100 = 100 customers

Step 2: Target Conversion Rate

How long would it take to reach this goal? Let’s evaluate site traffic. Suppose your client’s Web site receives 10,000 visitors per day. If 50 of 100 visitors who start the booking process complete the process successfully, the conversion rate is 0.5%.

50 people completed the booking process / 10,000 site visitors = 0.5%

Your client may have told you that 20% of those who complete the process are unpaid bookings—thus, they don’t increase sales revenue. After excluding these orders, the conversion rate is 0.4%, or 40 orders a day.

The client’s current Web site generates $4,000 of revenue a day. Increasing its conversion rate by 1% would increase revenue to $10,000 a day. Therefore, a UX design effort that achieved this would enable the client to earn $6,000 more in sales revenue per day, which would pay off their UX investment in less than 2 days.

($10,000 - $4,000) / $10,000 = ~2 days

Step 3: Using Conversion Rates to Communicate the Value of UX

The benefit of hiring UX designers seems evident, but you can go further and compare the results of an improved user experience with other ways of acquiring customers. Your client is already trying to grow sales—whether through advertising, a deferred activation on-demand program, or some other marketing approach. Let’s assume that monthly advertising expenses are $100,000. In comparison, the development budget is only 10% of that, and this is one-time expense. If your client reallocates a part of their marketing budget or expands the development budget for just one month, they’ll be able to increase the efficiency of their monthly ad spending thanks to increased conversion.

Customer acquisition costs through advertising are greater than through other channels. Assume it’s $50 for advertising. This is much greater than $10—the average cost of customer acquisition through all channels. Increasing the conversion rate makes advertising more efficient—more newly acquired customers pay for their orders—so the client’s customer acquisition costs have decreased to perhaps $40. In turn, this means that the client can acquire same number of customers with less monthly ad spending. If it cost $10,000 less, the client could spend this money to reinforce their marketing effort—for example, by ?advertising through additional channels.

Limitations of Conversion-Rate Metrics

These calculations look good, but the reality is a bit more complicated. It may not always be possible to get accurate figures, and increases may not be so large. Plus, this hasn’t taken the client’s other marketing efforts into account—for example, special offers or price changes. And there may be other factors like seasonality, the differing efficiency of various traffic sources, and deferred demand. But your efforts can raise conversion rates.

However, bear in mind that, with services like social networks, increasing the number of registrations doesn’t directly increase profits. So you’ll need more complicated calculations. For ecommerce Web sites, it’s easy to see a direct connection between conversion rates and growth in profits.

How to Obtain and Validate Conversion-Rate Metrics

Where can you get all of this information? Part of it comes from Web analytics. Google Analytics is a free tool that helps you to track conversions and learn about user behavior patterns. You can get many figures and a lot of other information directly from your clients—chiefly from their CRM (Customer Relationship Management) systems.

To understand and configure a client’s conversion funnel, you must understand its usage scenarios. Communicating these through a diagram lets you identify all key points in the funnel and find bottlenecks, enabling you to answer these questions:

  • What pages must a customer visit during a checkout process?
  • How does a customer start the purchasing process—coming from what traffic sources and landing on what pages?
  • What happens when a customer completes this process? Does he receive a confirmation email message or phone call? Must he visit a company’s office?

For a conversion optimization project to be successful, your client should implement and test all potentially viable UX design solutions. For almost any problem, there is a range of potential solutions from which it is possible to choose the most suitable one only by testing all of them in actual use by customers on the site. You can run A/B or multivariate tests to compare the performance of design alternatives, then choose the best one. This approach quickly gives you statistically significant results.

My final advice is not to chase after the one and only killer solution that would immediately multiply conversion rates. In practice, there are dozens of small tweaks you’ll need to make to achieve optimal results.

Metric #2: Average Revenue Per User

For both subscription-based applications and services and those that use a freemium model and rely on regular user participation, the average revenue per user (ARPU) metric is highly important. The goal is to motivate customers to use more product features and, in this way, grow your client’s profits by selling more services and products. For a SaaS (Software as a Service) product, this might mean motivating customers to purchase an enhanced user account; for a massively multiplayer online game (MMOG), getting players to buy more artifacts.

If you can prove that user interface changes would help to increase sales to existing customers, your client will be much more willing to listen to your reasoning and increase the UX budget.

How to Calculate ARPU

Let’s take a productivity application for small businesses as an example. The key metric for such a Web application is average payments for service usage per user. To calculate ARPU, you must divide all revenue for paid services by the number of registered users. Here’s the formula:

revenue from paid services / number of registered users = ARPU ($)

Step 1: Starting ARPU

As in the previous example for the conversion metric, let’s assume that initial research, a minor redesign, and its implementation would cost $10,000. Suppose that the product has 10,000 registered users, and each of them pays $5 a month on average for an enhanced account. Increasing revenue might mean selling additional services to earn $1 more per user. Alternatively, you might try to decrease the average cost of converting new customers—although this is not something to which a service’s user experience can contribute greatly.

$10,000 / 10,000 users = $1 per month

Step 2: Target ARPU

Once you know your target ARPU, you need to determine how to achieve it. For what services do you need to increase sales, and how much would it cost to achieve this additional revenue? Assume that you’ve examined your client’s business and have found that 10% of users would have to subscribe to an enhanced account rather than a standard account to gain a $0.75 ARPU increase. In addition, your client might increase sales of consultations by 10%. With an average price of $100 per consultation, that would increase revenues by $0.25 more.

$1 in target ARPU increase = $0.75 + $0.25

You need to understand what design changes would pay for your UX budget and how long it would take to pay for it. This is a matter of planning rather than precise calculations—that is, determining a set of activities that would lead to your achieving your target numbers in, for example, 2 months. In this case, your initial message has changed: if it would take several months to get to your $1 target, and your plan assumes that, in each of these month,s ARPU would grow by $0.33, your UX budget would pay off within 2 months. Here is the formula:

$1 → $0.33 per month #1 + $0.66 per month #2

Step 3: Using ARPU Metrics to Communicate the Value of UX

All the numbers add up, but why should your client invest in the UX changes if their product is already profitable? Because this is a matter of generating additional profits, communicating the time it would take to achieve a certain amount of profit is the best way to lobby for proposed UX improvements. But you can amplify the effect of this approach by providing a summary of revenues over a longer period of time—a year, for example. Acknowledging the gradual achievement of a $1 target for ARPU growth lets you show that the client can earn $110,000 in additional profits with the current number of registered users. And if their user base typically grows by 100 people a month, yearly earnings growth would be even greater—$116,567. The formula is:

ARPU per year = ARPU per month #1 * number of registered users in month #1 + … + ARPU per month ## * number of registered users in month ##

After subtracting the UX budget, the client’s additional profit would be $106,567, and they would undoubtedly have business development plans that would benefit from this unexpected funding. Your getting work on such a project becomes much more feasible when you bring these kinds of numbers to contract negotiations.

Limitations of ARPU Metrics

I’ve simplified my description of this metric in many ways. If a client proposes selling additional consultations, you may need to take other considerations into account—for example, possible limitations in the availability of specialist resources; new tariffs that could significantly increase the workload of a support team; or decreased user loyalty resulting from a company’s motivating customers to switch to an enhanced account by limiting the capabilities of the cheaper version. But even a simplified justification for your UX budget can improve your negotiating position. And you can use a more precise metric whenever possible.

Be careful not to break something that’s already working. You can quite easily revert to your previous design if a new design doesn’t achieve the expected results. Doing this won’t affect new orders. But you’ll need to recover the loyalty of existing users if a new design does not work well.

How to Obtain and Validate ARPU Metrics

Increasing ARPU is about generating more revenue from existing customers who are already loyal users of a product. Through user research, these users can help you to understand and meet their needs. You need to know what motivates people to use a service, and those who are already using it can tell you. You must identify the barriers that prevent people from successfully using a product, and people who don’t use a service can help you to identify them. You can find out what you need to know by running a series of qualitative and quantitative studies—for example, interviews or surveys.

Google Analytics and other Web analytics tools tell you what the real usage scenarios are for a service. How are users landing on a site? How are they using it? How does a service connect with other task scenarios? At what point did a user make payment? What other services does a user employ? You also need specific numbers regarding payments and can obtain these from your client’s internal accounting system.

Metric #3: Support Costs

Any commercial product presents post-sales liabilities to your clients. First among these is the cost of providing support services. A company must help users to set up and learn their product, solve usage problems, and go on site to fix complicated issues. Doing all of this involves overhead costs that your client wants to keep to a minimum. But they want to minimize their costs without damaging user loyalty. People calling in for help don’t want to hear: “An operator will answer your call in 20 minutes.” In addition to phone support, support services include responding to incoming feedback-form messages, providing support via online chat, managing support forums, and field staff visiting users on site.

If you can show that UX enhancements would help to decrease the number of support calls a client receives, it becomes much more likely that the client would approve your UX budget.

How to Calculate Support Costs

Looking again at a productivity application for small businesses, the key metric is average support cost per user. Thus, you’ll divide all support costs by the number of registered users. Here’s the formula for calculating support costs per user:

total support expenses / number of registered users = support cost per user ($)

Step 1: Starting Support Costs

Again assuming that initial research, a minor redesign, and its implementation would cost $10,000, suppose that a product has 10,000 registered users and support for each of them costs $2 per month on average. Your goal might be to decrease that number by $1 per user—the opposite of the goal for the ARPU metric, but again making calculations per user.

$10,000 / 10,000 users = $1 per month

Step 2: Target Support Costs

Once you know your target support cost, you need to figure out how to achieve it. What UX improvements would significantly lower the number of support calls your client receives and thus reduce the budget necessary for support? For example, you may have examined your client’s business and found that decreasing the number of questions about setup and learning by 25% would allow your client to save $0.6 per user. And fixing a Web form’s legacy usability issues would reduce the number of support phone calls by 5%. That would give your client a savings of $0.4 per user.

$1 target support decrease = $0.6 + $0.4

Now that you’ve identified the user interface changes that would reduce support costs and help pay for your UX budget, you need to calculate the time it would take to get to the target number. This depends on how fast you can fix the usability issues. If users’ troubles with the Web form disappeared right after its relaunch, you’d achieve the desired return on investment in the first month. On the other hand, a reduction in the number of questions about setup and learning might occur gradually—say by 50% a month—so at the launch of the redesigned user interface, you’d achieve a savings of $0.3 per user. In this case, the reduction in support costs would pay for your UX budget in the second month. Here is the formula:

$1 → ($0.4 + $0.3) per month #1 + ($0.4 + $0.6) per month #2

Step 3: Using Support-Cost Metrics to Communicate the Value of UX

Although these calculations obviously speak to the benefits of an enhanced user experience, clients’ level of disbelief is often high when UX professionals talk about economy metrics. They’re harder to understand because many different parameters affect them. And clients may be less psychologically comfortable relying on them because they’re less tangible than profits. Therefore, it’s helpful to express these numbers in a way that is more specific and understandable by your clients—for example, the opportunities that freeing up money presents. To do this, you should calculate the total savings over a longer period of time—a year, for example. If you know a client’s business well, you can propose making previously postponed UX changes to achieve this savings. The formula is the same as for the ARPU metric, as follows:

economy per year = economy per month #1 * number of registered users in month #1 + … + economy per month ## * number of registered users in month ##

By gradually achieving the $1 target decrease in support costs, your client would save $117,000 within a year, assuming the current number of registered users. If, every month, the number of product users increased by 100, the yearly savings would be even greater—$123,600. Subtracting the money your client spent on the UX redesign effort gives an additional profit of $113,600. Communicating such a specific and tangible number to clients makes them much more comfortable with negotiating a UX budget with you.

Limitations of Support-Cost Metrics

I’ve simplified the calculation of this support-costs metric in many ways. What if the UX changes you want to make are so pricey that it would be cheaper to hire a dozen new call center agents? Or what if an improved user interface would result in the firing of some support agents, so they would sabotage your new user interface? Plus, many stakeholders like to solve problems with call-center loads by adding to a Help knowledge base, leaving the product’s user interface unchanged. In any case, this metric is still useful as one way of persuading your clients and justifying your UX budget.

How to Obtain and Validate Support-Cost Metrics

Your client can supply data on the number of support calls they receive per day and the reasons for those calls. This data can help you to discover what problems you need to solve first. However, this data alone can’t give you sufficient detail about the problems users are experiencing. First, it’s not easy to understand exactly what prompted a user to call and make a support request. Though you can track this better via an online chat. Second, the problem may be with the site’s content rather than its user interface—for example, an ecommerce Web site might not provide information about shoe size and color.

There aren’t many precise numbers you can use in deriving support-cost metrics, so your main task is to find usability problems you can solve. You may not be able to talk with customers directly to get this information, but support agents have a lot of the information that you need. You could run a series of interviews with them or observe their work for a couple of days and hear about the problems users report first hand. It would be even more helpful if you could work as a support agent yourself for a couple of days, but most call centers have pretty tight policies restricting who can take calls. You could also try recruiting support agents to do user research for you—for example, by providing additional questions for them to ask users. But, if a call center is already overloaded, it would be difficult to get your client’s approval for this.

Metric #4: User Performance

When users work with a product on a regular basis and repeatedly perform the same operations day after day, optimizing these operations is always beneficial. While the loss of 10 seconds by a single user during each workday would not be readily apparent, if you combine the time lost by all users who perform an operation over a span of many days, a single inefficient operation can decrease both a company’s efficiency and its profitability. Examples of this might be a cashier working with an old-model cash register or a bookkeeper who struggles working with a banking application that has an unusable user interface.

If you can demonstrate that UX enhancements would decrease users’ execution time for key operations and thus save a company money, you’ll find it’s easier to get a contract for UX redesign work.

How to Calculate User Performance

Let’s take users of a modern cash register as an example. The metric we’ll look at is speed of execution for the most common operation: producing a receipt.

time to execute operation #1

Step 1: Starting Performance Rates

Let’s again assume that initial research, a minor redesign, and its implementation would cost $10,000. Suppose a company employs 100 cashiers, an average receipt includes 10 items, an average total for a receipt is $50, it takes 5 minutes to create this average receipt, and a cashier creates 100 receipts per day. Salary and other expenses per cashier account for $1,000 a month. If every month includes 160 working hours and 20 working days, the cost of each minute a cashier works is $0.10.

$1,000 / 160 hours * 60 minutes = $0.10 per minute of a user’s work

This would require a time savings of 1,000 minutes per cashier every month to cover the UX budget.

$10,000 = 100 users * $0.10 per minute of a user’s work * 1,000 minutes

Step 2: Target Performance Rates

To achieve your target for this metric, you would need to reduce the time it takes to create a receipt by 30 seconds per operation, meaning the average execution time would have to drop to 4.5 minutes.

1,000 minutes / 100 receipts * 20 days = 30 seconds

How can you achieve this productivity increase? You could do this in either of two ways:

  • experiment—Do user research to find bottlenecks and determine whether user interface tweaks would affect users’ work performance.
  • analyze GOMS (Goals, Operators, Methods, and Selection rules)—

    Decompose the operations you’re measuring into discrete actions and determine whether you can eliminate or simplify some of them. Despite this method’s limitations, you can use it to find bottlenecks in a sequence of user actions.

Here is the formula:

total time to execute an operation = time to execute an action #1 + time to execute an action #2 + … + time to execute an action ##

While obtaining the data you need to calculate this metric, you’ll become aware of redundant and time-consuming actions. You can simplify certain steps or even eliminate them from a workflow. And your data validates that your UX design decisions are correct.

Step 3: Using User-Performance Metrics to Communicate the Value of UX

One of the best ways to provide a rationale for your UX budget is to show how much money your client would save over a long period of time—a year, for example. In this case, a savings of $1,200,000 would translate to a $1,190,000 savings after subtracting the UX budget. And if you know your client’s business well, you’ll likely be aware of another long-postponed optimization project to which your client could allocate this money, which would, in turn, save even more money for the company.

economy per year = economy per month #1 * number of users in month #1 + … + economy per month ## * number of users in month ##

Limitations of User-Performance Metrics

Similar to the support-costs metric, there are many real-life limitations that can complicate the use of this metric. In the case of our example, it might be cheaper to employ more cashiers than to redesign a complex product. Considering that the need to optimize the efficiency of a product’s operations typically arises when designing feature-rich, complex products—for example, enterprise systems, technically complex programs, and tools for tricky business processes—you may need to consider dozens of factors. There might be numerous technical and political constraints that limit what a UX designer can achieve. Plus, it can be difficult to obtain the average execution time for some operations because of the varied types of orders that cashiers work with in different departments; credit-card payments can take more time than those in cash; and the average number of operations can change from day to day. Finally, the overall user interface may need a complete redesign rather than just optimization. However, having numbers that support your suggested changes can mean more to your clients than your just having interesting ideas.

How to Obtain and Validate User-Performance Metrics

How can you measure users’ work performance? You can run two usability studies—one study at the beginning of a project to obtain your initial data and establish a baseline, then once developers have implemented the UX changes, another study to prove their effectiveness. It’s best to conduct these studies in the natural environment of a user’s workplace rather than in an usability laboratory—which would necessitate your also running separate ethnographic studies. First, you can avoid the bias of users’ working in an unfamiliar environment. Second, you can carefully study and consider a user’s context of work. In addition to calculating specific metrics, you can create detailed usage scenarios based on your understanding of users’ work. These can help you to identify bottlenecks.

Such user research should take into account several considerations. You should measure the speed of users’ work both in normal situations and when users are pressed for time—their behavior can differ significantly in these two circumstances. Plus, experienced and novice users may perform work differently, so you should consider the impact of experience on users’ speed of execution. Also, users may work on different systems, and you should understand how each of these affects their productivity.

Metrics in Practice

The calculation of metrics like these has helped me a lot in my work. Sure, there are lots of clients who you won’t be able to convince even with all of these statistics on hand. Many clients don’t believe in doing user research. Some may think the users you’ve recruited are the wrong kind of users—even if you’ve recruited them properly. But clients who understand their own business and how to increase its profitability would find these calculations both familiar and comforting.

Metrics let you measure the value of your work as a UX designer not in the abstract—the intangible value you’ve provided to mythical happy users—but in terms of the profits your work delivers to your clients and their customers. Of course, the people benefitting from your work include users, too, because your improving all of the aforementioned metrics relates closely to a product’s being both useful and usable.

Things get even more interesting when you employ several metrics on one project and thus achieve synergies between them. For example, you could both increase the number of registered users who pay more for additional services and give them fewer reasons to contact support. If you’ve had opportunities to do this, I’d be interested in hearing how it has worked for you in practice. 

Thanks to my colleague Natalia Sprogis for her advice and contributions about the user research process.

Head of Portal UX at Mail.Ru

Moscow, Russia

Yury VetrovYury leads a team comprising UX and visual designers at one of the largest Russian Internet companies, Mail.Ru, which is part of the Mail.Ru Group. His team works on communications, content-centric, and mobile products, as well as cross-portal user experiences. Both Yury and his team are doing a lot to grow their professional community in Russia.  Read More

Other Articles on Business of UX

New on UXmatters