UX Analytics, Part II: Getting a Quick Win

Narrow In

More often than not, your first pass is a springboard for more questions that would then lead to your gaining actionable insights. You and your partner might still have another 45 minutes or so of your one-hour meeting. Therefore, ask your partner to add one of the subflows to the funnel to see how it alters the data. The same questions I asked earlier still apply, but now there is an additional layer of questions around the specific test case that the flow reflects. What new insights did you gain from this deeper examination? Does it support or shed light on any of your earlier hypotheses?

Spend the rest of the hour plugging in various subflows and variations of subflows to see what interesting patterns appear and help you evolve—or kill—your hypotheses.

What’s Your Story?

From the above exercise, you’ve gathered data and built some hypotheses. Perhaps you want to let your hypotheses simmer over time to give yourself the opportunity to gather historical data for trending or internal benchmarking. Or, perhaps you want to gather data from alternative yet complementary data sources—such as VOC (Voice of the Customer) reports, customer satisfaction survey comments, call-center reports, or usability testing—that would enable data triangulation or data synthesis for even more fine-tuned insights.

What recommendations might you make based on your hypotheses? Your recommendations could be as small as changing the copy for a call to action or as big as redesigning and re-architecting the logic behind a form. Or, your recommendations might suggest that you should conduct further user research such as moderated or unmoderated remote usability tests.

As an aside: The closer you are to understanding the level of effort that implementing your recommended changes would require, the better prospects you have of making a quick win.? Some recommendations—like re-architecting the sign-in process—might require a team of people to examine the current implementation in detail, then determine a better solution to the problem. Other recommendations might be simple to implement—for example, tweak error message copy to clarify what a user can do to recover from an error. Both options are perfectly viable, but the more awareness you have around level of difficulty means you can be that much better at setting expectations around when stakeholders might see results.

What’s the Impact?

At some point, you may be prepared to claim that, if your recommendations get implemented, you could, for example, decrease the abandonment rate on your site or increase the conversion rate for a given process or step. The level of rigor you can achieve around your projections of how much you can improve on these rates depends on your organization and its readiness to make rapid changes—or reverse changes if they produce undesirable effects. For small, agile organizations, a let’s-try-it-and-see approach should work fine for enhancements that cost very little effort to implement, assuming you follow up with post-implementation analysis to ensure the changes haven’t had a negative effect.

For larger organizations—even on your first attempt—you may want to partner with a trusted internal statistician—who might be your trusted Web analytics partner—to build out a revenue-projection model. For example: If we decrease abandonment at login by 25%, we may realize an additional $300K in revenue. This becomes increasingly important as a solution becomes more difficult to implement.

Re-evaluate

Once changes are implemented, again measure the data for the business process funnels and compare the new data to your previous measurements, making sure to examine the same sets of days both before and after making the changes—for example, seven weeks prior to and seven days after making the changes, with both sets of data running from Sunday through Saturday. And, make sure there weren’t other changes to the business process flow that you’re measuring or, if there were, provide some form of rationale in your methodology to account for those changes.

Conclusion

Funnel analysis can be an excellent first step toward eliciting some data regarding users’ experience and the impact of your recommended changes to that experience. As designers, we sometimes perceive the reasons for these abandonments as obvious, because we recognize that little things can be a big deal and cause customers to abandon. The challenge lies in understanding how severe issues are, measuring their impact, and projecting their potential, with the goal of getting those in leadership positions to take note, carry your issues forward, and instigate action.