This is the first edition of my new column, Data-Informed Design, which will explore the use of data to inform UX design. Data, in many different forms, is changing how we think about ourselves and the world. And, for better or worse, it is definitely changing our experience with technology—from great new mobile apps that we can use to monitor our health to incremental improvements on our favorite Web sites to those annoying ads that follow us everywhere.
In my column, I’ll describe how to use different types and sources of data to create better user experiences and how to achieve some balance—so data isn’t driving decisions. There are three key topics that I’ll cover:
Starting at a high level, I’ll look at why you would want to use data, some misconceptions around data and UX design, and discuss a process for incorporating any kind of data into your decisions.
Then, I’ll move on to considering various data sources such as analytics, A/B tests, social-media sentiment, and various types of quantified data from UX research.
I’ll also describe how to use and analyze data, including metrics and measurement frameworks, as well as presentation tips and visualization tools.
Getting Started with Data
About two years ago, I was on a call with a client and asked how he was doing. He told me, “Actually, I have a lot of data about that.” We went on to compare our sleep patterns with data from our self-quantification apps. It made me realize how much data is becoming part of the fabric of our everyday lives and how we expect data to make our experiences—in both real life and online—better.
Using data in design may seem daunting—or even just unappealing. When we hear about data in the news, it’s usually about surveillance, selling, or maybe self-improvement. When we hear about data in our organizations, it’s usually about who is right or wrong and by how much. Yet, we know that the future belongs to those who understand data—or something like that. Big data gets hyped almost as much as the Internet of Things, and it’s at least as hip and cool as user experience.
Fortunately, one doesn’t have to be a data scientist or Bayesian statistician to get meaning out of data—and this is the topic of my first column. Data doesn’t provide all the answers, but it does provide worthwhile insights. I’m puzzling out how to use data to inform user experience. I’m sure many of you are, too. Together, we can learn to make sense of it. My hope is to help all of us to be more open to the myriad large and small opportunities for data to make user experiences better.
We’ll look at big data and design, the ins and outs of using different types of data sources, how to work with quantitative data and blend it with qualitative data, correlation versus causation, and of course, metrics. To get started, let’s look at why we would even want to use data in UX design at all.
Why Bother with Data?
For any Web site or application, it’s typical to do analytics, A/B tests, social-media sentiment, surveys, intercepts, benchmarks, and scores of usability tests, ethnographic studies, and user interviews. We end up with a lot of data. There is much debate over which data is most meaningful and hand-wringing over how our organizations can connect all the dots. But at its core, data about Web sites and applications is data about the people who use our products.
So, for me, the traces that the people who use technology leave behind—no matter their source—are data that we can use to inform UX design. Just as we wouldn’t look at an archaeological dig and expect to create a complete picture of life in Ancient Rome, we can’t look at any dataset and create a complete picture of the user experience. Data represents an approximation of the user experience, not a matter-of-fact truth.
A big part of using data to inform UX design is figuring out what we care about most. When UX designers use data, I’ve found that there are really three main stories:
Proving a point
Improving an experience
Discovering something new or, at least, something that’s not obvious
For better or worse, UX professionals often use data to prove which choice is right or wrong—whether to solve battles with stakeholders or counter a gut-feel approach. For example, proving return on investment (ROI), or which variant of a design would be more profitable, is a valid and common use of data. Proving a point, at its best, is really about determining the right path to success.
Let’s say your team wants to show that a design change is responsible for an increase in signups. You could simply look at how many people visited a site in relation to how many people signed up since your team redesigned the site and make that claim. Of course, there are reasons for people to be a little suspicious of data and how it connects to design decisions. First, there are many variables that factor into signups. Second, this metric is too high level to tie back to specific design improvements.
Perhaps, instead, if there were specific data that mapped to a particular design change, that would make a better story. For example, designing an A/B test to see which of two sign-up pages produces the better result is another way to use data to prove a point. A/B tests are great at showing which design decisions have the most impact. On the flip side, A/B tests are very granular and, at a certain point, you have to step back to see the big picture.
In the end, proving a point with data is not clear cut. You can use one dataset to prove that cats are more popular than dogs on the Internet and another to prove the opposite. Typically, the data that UX designers use to prove a point is data that comes with ready-made metrics such as those from analytics tools. This is not always the most relevant or meaningful data.
The bigger the idea, the harder it is to tie it back to one data point—or even one dataset, or collection of data. But if you don’t understand how to use data to prove a point or you’re not comfortable working with multiple data sources relating to a user experience, you’re at a disadvantage.
Another way you can use data is to show improvements in a user experience—typically, by quantifying a data point that you want to improve and tracking it. There are plenty of examples that have nothing to do with online user experiences. For instance, the Chief Analytics Officer of New York City—yes, there is one—has used data science to reduce ambulance response times there.
Now, let’s look at our signup example again. If you wanted to see how much you’ve reduced the amount of time it takes to sign up, you could look at analytics to determine that. Yet you would still have a big question: why? So you would also need to conduct usability tests to determine why that had changed and what you could improve further.
You don’t have to start with analytics data to improve a user experience though. Let’s say that, instead, you wanted to understand how people perceive the ease of use of the signup process. You would need to develop a questionnaire to learn about that—preferably using something like the System Usability Scale to see how much it has improved over time or how that perception matches up across all touchpoints.
The key to using data to improve a user experience is a lot of consistency in how you measure things, plus a little experimentation. Experimenting can help Web sites like Etsy to make incremental improvements that add up to a better user experience. Measuring in the same way—over time or across sites—can help you to see changes and, hopefully, improvements in a user experience. So, whether we want to see how dogs show steady improvement in their quest to beat out cats as rulers of the Internet or to understand how to improve key aspects of a Web site’s design, data from a variety of sources can help us to make the user experience better.
Discovering is the most interesting story, but it is also the most complicated data story. For example, whether we look at Target’s ability to predict that a woman is pregnant or Facebook’s questionable social experiments on unwitting users, there are certainly questions about the user experience that can be answered by looking at relationships between data points or among datasets. That is what discovery is all about.
Going back to our signup example one more time, maybe we want to see whether there is a relationship between the actions users took before signing up on a site. This could be an analytics question about where they came from or even how much they read, depending on what analytics tool you are using. Social-media data analysis could also answer this question by looking at conversations around signing up on the site.
In the field of user experience, we are most familiar with using qualitative research studies to discover new insights. We could answer this same question about signups through an ethnographic study examining the journey to sign up and the context around it. Qualitative studies are data, too. Smaller datasets and convenience samples—whatever participants happened to be available to a researcher—are typical of behavioral economics research and have led to major breakthroughs in that field, but that data is doubly useful once it’s coded and tallied.
Algorithms can answer some questions, but generally, data does not speak for itself. For example, to understand the subtleties of the Internet cat and dog conflict or discover what behaviors precede a sign-up process, you need to be constantly asking questions—the most important one being: So what?
Defining the Thing
There is a lot of buzz about data-driven or data-informed design, but very little agreement about what that really means. The data that we use to inform design goes beyond analytics and A/B tests, spanning behavioral tallies, textual analytics, and data from studies of all kinds.
One thing we do know is that the more closely connected data is to the experiences of people and how they perceive them, the more meaningful and actionable that data is. The challenge is asking the right questions of the data to deliver insights that are relevant to the user experience.
In future installments of my column, I’ll cover topics such as the following:
What counts as data for user experience?
Thinking with data
Aligning business, marketing, and UX data goals
Metrics, KPIs, and scores
Qualitative versus quantitative data
Where to start with analytics
A/B testing for UX professionals
Social-media data and user experience
All about conversion metrics
Usability scores and measures
But this is by no means an exhaustive list, so please comment if you have other topics or questions that you’d like to explore.
Pamela is founder of Change Sciences, a UX research and strategy firm for Fortune 500s, startups, and other smart companies. She’s got credentials—an MS in Information Science from the University of Michigan—and has worked with lots of big brands, including Ally, Corcoran, Digitas, eMusic, NBC Universal, McGarry Bowen, PNC, Prudential, VEVO, Verizon, and Wiley. Plus, Pamela has UX street cred: She’s logged thousands of hours in the field, trying to better understand how people use technology, and has run hundreds of UX studies on almost every type of site or application you could imagine. When she’s not talking to strangers about their experiences online or sifting through messy data looking for patterns, she’s busy writing and speaking about how to create better user experiences using data of all shapes and sizes. Read More