Top

Death by Micro: Feedback Loops and Knowledge Management in User Experience

May 31, 2016

In the evolving world of knowledge-driven organizations, we are both blessed and cursed by the vast amounts of data that can help influence our opinions and shape our actions. The blessing is our ability to more openly and immediately access data that can help us in many ways. On the flip side, the curse is the subjective manner in which people sometimes use data as proof to warrant an action that might be problematic. I’ve personally had many professional and personal experiences that speak to this reality.

Micro Research

A common theme in my experience as a UX Designer is that measuring an interaction or a particular experience in a vacuum can be misleading. Even if you measure an experience using the proper methods, the perfect sample size, and all the right tools, you still might find that you’ve done yourself and your product a huge disservice. The world is chaotic and, when we make the mistake of sinking down into the abyss of overly minute investigations into micro interactions, we lose sight of how the way users got there and where they’re going next affects the overall experience. We lose the context. In the knowledge era, this type of knowledge management has become vital to organizations.

Sponsor Advertisement
Continue Reading…

One personal example from a previous work experience comes to mind: I was working for a social-media curation company that was really keen on increasing share rates for their articles. So, naturally, they did quite a bit of A/B testing on how the size, placement, color, and other variables for the Share button affected the number of shares they were able to squeeze out of users. Because social sharing was a key metric that drove their overall profitability, many employees might have viewed pursuing this particular research effort as a successful endeavor in and of itself. But I disagree.

Macro Research

Our not taking a macro look at the overall user experience by doing more comprehensive user research was a lost opportunity. Neglecting other means of measuring the user experience is a potential downfall of any very specific research method such as A/B or multivariate testing.

It is important to ask macro questions such as “Where are users getting stuck?” and “Do users understand our value proposition?” so we can frame micro-focused questions such as “How can we get users to share more?” relative to larger user-experience and business goals.

Looking at only a single metric such as users’ inclination to share articles is not necessarily indicative of the quality of the overall experience they’re having. How would users feel about a more ostentatious Share button after 5 minutes of use? After 2 weeks? These were questions we didn’t, but should have asked up front.

After the Share button redesign, while the company experienced an initial, positive boost in some KPIs, the repeat interaction rate ultimately dropped, and the longevity of our user base diminished. Collectively, these negative metrics ended up weighing more than the number of social shares we got early on. Taking an overly narrow, micro viewpoint in both research design and the validation of research can be dangerous.

Supporter Advertisement
Continue Reading…

The Impact of Feedback Loops

Even more dangerous is the possibility of our not connecting the dots, failing to understand our mistake, and reflexively iterating on it in pursuit of future success. Believe it or not, this happens a lot! But why does this occur? Because of organizational feedback loops!

The concept of a feedback loop is pretty simple. Feedback loops provide a framework for interpreting how and why the feedback an organization receives on a user experience has influenced or failed to influence the way particular individuals reflect on a situation or solution. Transitively, feedback loops project a shadow of the future for similar interactions.

Applying the concept of a feedback loop to the scenario I previously described, how might the internal stakeholders of that social-media company have interpreted what happened and, more importantly, how might their interpretation of the data affect the design of similar interactions in the future?

Seeking a Different Outcome

The internal stakeholders might have enabled their interpretation of the data to impact long-term outcomes more positively. That is, rather than focusing on the immediate results—an increase in social shares—they might instead have analyzed the overall, long-term experience outcome—a frustrated and diminishing user base. They could then have tied that feedback back to their having made a design decision based on a single research method—in this case, A/B testing.

Looking at this situation optimistically, this might have persuaded these stakeholders to have a change of heart regarding their decision to implement in-your-face sharing features throughout the user interface. Even more importantly, this change in perception could have positively affected how stakeholders within the organization choose to design future research.

On the flip side, the distance in time between the A/B test and receiving the actual negative feedback—the weakening of certain KPIs—may have resulted in a missed learning opportunity for both the individual and the organization. When such events are spaced too far apart over time, this can—and often does—lead to a team’s inability to accurately perceive cause and effect. This is particularly likely in regard to organizational KPIs resulting from particular design choices.

Conclusion

In this new era, when organizations place so much emphasis on knowledge management and making data-driven decisions, UX professionals, our stakeholders, and our peers on multidisciplinary teams must bear new responsibilities. One of our new responsibilities is opening up our minds to analyzing information by looking at the long-term, macro view.

Organizations’ ability to avoid falling victim to the potentially negative consequences of feedback loops will strengthen or weaken organizations as a whole,. Missing or misinterpreting information can result in the design of bad user experiences in the near term. But, more significantly, an incorrect viewpoint on how to incorporate user-empathy methods into product design can lead to, not just one, but many design flops over the life of a product.

Capitalizing on opportunities to connect user feedback to the appropriate decision or action can certainly be a challenging goal, but achieving that goal is one of the most rewarding outcomes of UX research and design. 

UX Design Consultant at Slalom Consulting

Chicago, Illinois, USA

Dashiel NeimarkDash designs usable, enjoyable digital experiences that are driven by research and guided by the needs and desires of internal and external stakeholders. In his work, he draws upon his past experience in startups, UX consulting, internships, and freelancing, as well as the wealth of UX knowledge he gained through his journey to earn an MS in HCI. From concept to launch, Dash incorporates Lean and full-cycle UX tools and methods. He is always excited by future opportunities to play his part in delivering innovative digital solutions.  Read More

Other Columns by Dashiel Neimark

Other Articles on User Research

New on UXmatters