Introducing the Build-Measure-Learn Approach to an Analytics Tool’s Redesign

Being Careful About What We Asked

To determine what reports to prioritize and what data they should comprise, we decided to create an in-app survey that would have a two-fold mission:

1. Gather more feedback directly from customers and users.
2. Develop a list of customers that we could reach out to during the design phase.

When a user visited the Analytics page and some trigger criteria had been met, this survey appeared within the application. Our survey asked a single, open-ended question: “If you had to name one thing to change on the Analytics dashboards, what would that be?”

The usefulness of the answers that we received to our survey question was somewhat mixed. The more helpful answers validated some of our initial assumptions—such as the need for more flexibility on date ranges and the ability to view a community’s entire history. A few data values that we hadn’t expected to be useful also surfaced. But the less helpful answers were either too vague or seemingly contradictory—for example, “More clarity,” “Cram more data into a single screen view,” or simply requests for “More data!”

A huge advantage of our survey was that it collected participants’ email addresses—which correspond to user names—helping us to build a list of customers to whom we could reach out. On subsequent projects, we’ve found that our in-app surveys give us access to a broader sample of customers who use a feature—not simply a list of the most vocal customers or people our team has contacted recently.

Scheduling Time for Experiments

There are many ways in which we could interpret “more”—especially given the different ways in which our customers use the product—so there was risk that our guess would be wrong. How were we going discover the right balance between providing more details while sticking to our mission of simplicity? After our initial analysis of the survey results, we decided to run two rounds of qualitative experiments:

1. Validate that we had included the right data in the reports.
2. Test a user interface that would let users find and select their reports.

So, over a two-week period, we set aside two days a week for remote usability tests. The first week, we ran the first experiment; the next week, the second experiment. We invited customers who had responded to the Analytics survey to participate in a half-hour usability study, using GoToMeeting. Customers chose the time slots that worked best for them over the four days available. More than a dozen customers volunteered their time to test drive our design ideas and shared invaluable feedback.

Validating Our Assumptions First

For the first week of testing, we designed an experiment to validate that we were creating reports that included the data points that our customers needed. We decided that, if we started by conducting usability testing on the spreadsheets, it would help us to know whether we’d chosen the right information—before we started designing the screen that let users select their reports. So we designed spreadsheets that represented the reports that users would download from the Analytics page. These spreadsheets, which mocked up three months of activity for a hypothetical community, addressed the use cases that we’d identified. We imported our mock data into spreadsheets in Google Sheets and linked it together in a Google Docs document.

At the beginning of each virtual test session, we asked users to think back to the last time they needed to answer questions about their community—whether they referred to this data regularly or occasionally needed answers to questions that they weren’t quite sure how to answer. We then asked them to tell us about their three most common usage scenarios. If we had prepared a spreadsheet that addressed one or more of the scenarios that they described, we directed them to the appropriate spreadsheet and used their scenarios as their test tasks. (We chose to limit each test session to three tasks so we could stay within the planned session duration.) But we had lists of up to five possible tasks that we could choose to test—in case their scenarios were edge cases or we hadn’t prepared a spreadsheet for a scenario.

We instructed participants to pretend that they’d downloaded each spreadsheet from their Bloomfire site and were viewing them offline. We asked them to read through each of the column headers in a spreadsheet, then describe what each label meant and what data it represented. We learned that some of the column labels that we had chosen were confusing—or meaningless in the context of users’ mental model of Bloomfire. For example, we learned that using the label Consumption to identify how often community members had read a contribution wasn’t as meaningful as the label Views.

Together, the open-ended questions that we asked at the end of each test session and customers’ sharing their screen with us during the virtual sessions gave us the opportunity to learn about some of the painful, circuitous routes that customers were taking to find data that we hadn’t surfaced in the Analytics tools. For example, several customers showed us the workaround they used to learn whether a new community member had viewed all of the content in a Series playlist.

Users can add the following types of contributions to their Bloomfire community:

• posts
• questions
• series, or virtual playlists of posts and questions

To determine whether a community member had viewed all of the contributions in a series, users were visiting each contribution in the series, choosing the Viewed details for a post or question, then searching for that member’s name in the list. They repeated this process until they’d visited each contribution in the playlist. Depending on the number of items in a series, this could be a very time-consuming workaround.

Iterating, Iterating, Iterating

Throughout the two days of testing, we used both users’ feedback and the stumbling blocks that we had observed during testing to inform design changes to the spreadsheet. Between test sessions, we iterated the design; then we tested again.

Once we were comfortable that we were showing the right information, our next experiment focused on designing a user interface that would let users select their desired report and easily move through different levels of detail.

Using what we had learned by testing the raw-data exports, we built a light-weight Axure prototype of a user interface that let users navigate between the reports. The purpose of the second experiment was to validate how to group the reports logically, how to navigate between different levels of detail, and what to label the groups. Much like in the first experiment, we were able to make small changes to the prototype between testing days to see which changes got the best results.

Well Begun Is Half Done

Weeks later, a few of the users who had volunteered for the earlier experiments are now participating in a limited beta program for the new Analytics tools. Our beta testers are already sending us feedback. We’re pairing that user feedback with regular calls to customers who are very engaged with Analytics. We’re also preparing traffic reports to monitor usage trends for the Analytics page to see what metrics change.

We look forward to sharing the first parts of the newly redesigned reporting features with our customers very soon. After doing several rounds of experiments, we’re confident that the design direction we’ve taken is many times better than what we would have designed without our customers’ help.