Top

Why UX Research Repos Fail at Democratizing Insights

April 25, 2022

Why are the products we use every day simply crap?

This was the question that our founding team at Airtime asked ourselves when we started the company to build our collaborative research platform, in late 2020. Thanks to rapid digitization in all sectors, a huge amount of data is readily available for product managers in any field. So why doesn’t this data leave a mark on the products they create? To answer this question, we did some desk research, signed up with mentors, and organized user interviews.

We found that the fields of user research and product design are growing by high double digits, along with the software market. But, now that user-research tools are many and growing in number and products are still not working well, we’re convinced that there must be something wrong with the current research paradigms.

Champion Advertisement
Continue Reading…

The Problem

The most popular research tools are currently research repositories, or repos. These structured data libraries house customer insights that companies have gleaned from their user research. They are searchable and the data is shareable across the organization. Such repos fit very well within the formalized research process that user researchers typically follow, and they help companies reduce the time and effort they spend on research over the long run.

The problem is: these repos are the end result of process-oriented research. Traditionally, research resides in a separate department that receives instructions from product managers who need to know more about their customer. Then researchers do their thing: they conduct interviews, organize surveys, and analyze data. Essentially, they gather input using different methods, then draw their conclusions, or insights, from their analysis. They then share these insights with the product managers, who use them in product development. The issue is that research has become a silo within the organization—not unlike a black box. Researchers receive their instructions, conduct their research project, and spit out the results.

According to Andreessen Horowitz, current user-research practices require long, expensive user-research cycles. Research is over tooled, manual, and prone to errors. Depending on the research method, feedback collection can take anywhere from days to months, incurring high costs in the five to six figures. This process, in turn, introduces budgetary constraints. So, in the end, companies conduct critical research projects only one or two times per year—which is way below the frequency at which product teams need them. Andreessen concludes that “the user research industry is ready for a new wave of innovation.”

Research repos are the result of these long, over engineered, discrete research projects. The data in them quickly becomes stale—assuming it wasn’t late in the first place. Unless teams update such repos frequently and at a high degree of granularity, they aren’t useful. Now, here comes the bombshell: even when companies build repos and maintain them meticulously, the companies still don’t use them.

Our Learnings

As we gathered real-world data and use cases for our startup, we kept hearing firsthand about the same problems from researchers and product managers. We conducted more than 200 interviews, and the problems we’ve outlined kept popping up in numerous discussions.

“Adoption of our new research repository didn’t go well because our colleagues didn’t want to create a user account,” said a UX researcher at a leading customer-engagement platform. User accounts were free, so all an employee had to do was to provide an email address and create a password, something we all freely do for a meager 10% discount on a nice-looking pullover on Farfetch.

The Design Lead at a petrochemicals company with ten client-facing products and fifteen internal products told us: “We have a lot of insights, but we are not using them…. Business decisions are made based on economics and technology, and insights introduce a new variable that makes everything more complicated.”

The Director of UX Research and Product Design at a company developing virtual presentation and collaboration tools clearly expressed his skepticism about insights repositories: “Repos are like a library, only 5–7% of the content is actually used. Compared to that, significant work must go into structuring the data, otherwise it will be useless. This is a big overhead.”

As different versions of the above snippets continued coming up regularly during our interviews, we decided we needed to look for an alternative research paradigm. An insight is valuable only if it makes an impact on the product. But how can it make an impact if no one looks at it?

The Key Factors of Successful User Research

McKinsey created a design index that has shown “the potential for design-driven growth is enormous in both product- and service-based sectors because the market disproportionately rewards companies that truly stand out from the crowd.” This translates into a 2X–3X multiplier in company revenues and shareholder total returns. (By design, they mean “the process of understanding user needs, … then creating products and services to meet those needs. This covers physical, digital, and service design, and is not just about aesthetics.” For the purposes of this article, I’ll use the term user research to refer to this field because it’s less ambiguous.)

According to McKinsey, the four most important qualities that drive an overall design orientation are the following:

  • analytical leadership—This kind of management approach looks at design performance as strictly as revenues and costs.
  • user experience—This is somewhat self-explanatory. Companies that excel at User Experience break down internal barriers between physical, digital, and service design and have a holistic view of where design can make a difference.
  • cross-functional talent—Successful organizations break down functional silos and make user-centric design everyone’s responsibility.
  • continuous iteration—Continuously checking in with clients and iterating a mix of qualitative and quantitative research is key.

(If you’re interested, you can check your company’s design index in comparison to those of over 300 companies by taking this survey.)

Thanks to McKinsey, we now have a blueprint of what an organization must do to excel in the field of User Experience. But this is easier said than done because the way in which companies organize user research needs to adapt. The formalized research-plus-repo approach goes against two of the four success criteria of McKinsey. As a separate silo, research can never be cross-functional. Neither can it be continuous when it focuses on discrete projects that are driven by ad-hoc product-management needs.

The Solution: Collaboration

The alternative is collaborative user research: involving internal stakeholders who are not part of the user-research team in the user research process—especially having them participate in customer-facing interviews and discussions. This solves many problems right off the bat—for example:

  • There is no need to share insights after the fact because the people who would be consuming them are hearing them from the client firsthand.
  • There is no delay in the information flow. Stakeholders hear the clients’ needs and problems when they say them out loud.
  • In-person interactions also build empathy toward customers and makes it harder for management to brush off the insights because of their effects on the bottom line.
  • Ideally, analyzing raw interview data and gleaning insights from them is also a collaborative process, so people in different departments and disciplines can interpret the input together. There is no possibility of separate interpretations because the team is aligned.

Jared Spool is a high-profile promoter of collaborative user research. He is a usability, software, design, and research expert, having worked in the field of usability and design since 1978—before the term usability related to computers.

In his article titled “Attaining a Collaborative Shared Understanding,” Jared differentiates between contractual shared understanding and collaborative shared understanding. In the contractual approach, clearly defined and documented tasks and deliverables such as wireframes, design guidelines, and user-interface specifications act as a recipe for the stakeholders’ implementing the designs. This is the formalized, old-school way of doing user research. The major benefit of this approach is that it’s good at predicting the resources that are necessary for implementation to create the final output. On the other hand, such manuals are open to interpretation and can create a mismatch between the intent of the designer—or user researcher—and the final execution. Manuals are also static. The collaborative approach doesn’t remove stakeholders from the design process, but instead keeps these two groups working in constant, close interaction with one another. They define the spirit of the design rather than the actual output and produce great designs at the cost of inaccurate predictions about deadlines and resources.

Jared argues that exposure hours are the solution. This is the time researchers spend in face-to-face meetings with clients and company stakeholders from outside research. He recommends spending at least two hours every six weeks in client-facing interactions. His views very neatly tie into the conclusions from the interviews my team conducted at Airtime, as well as McKinsey’s four factors determining design success.

To demonstrate how easy it is for an organization to implement a continuous, collaborative research practice, we’ve created a Collaborative UX Research Playbook that summarizes the key steps to setting up such a practice and lists some tools that make your life easier.

Conclusion

Leading corporations such as Atlassian are already using exposure hours daily in their research practice. Stakeholders such as product managers and engineers not only attend customer calls but can observe how customers interact with their product, ask them questions directly, and do postmortems after sessions with researchers.

It is incredible to see how the simple idea of direct client exposure kills five birds with one stone. It allows continuous design iteration instead of your having to run discrete projects. It lowers costs. It makes information flow almost instantaneous. It builds organizational empathy toward the client. It pushes management toward more customer-focused decisions.

All of this makes customers happier and more loyal, which drives revenues. If well-capitalized market leaders are embracing collaborative research, why wouldn’t you? 

Head of Operations at Airtime

Berlin, Germany

Peter VetoAt Airtime, a software company that develops collaborative user-research tools and helps organizations implement collaborative-research practices, Peter heads up operations. Peter and his two founding partners launched their user-research platform because they felt that many products—whether digital or physical, business to consumer (B2C) or business to business (B2B)—are less than optimal and could be improved through the simple exercise of listening to customers. Peter has a diverse background—from investment banking to institutional sales to working for asset managers, asset owners, and broker-dealers in Europe, Asia, and the Middle East. For seven years, Peter worked for the leading international provider of equity indices and investment analytics. Following his stint in the financial industry, Peter switched to freelance startup consulting and investment-analytics publishing. Peter has a Masters in Capital Markets from the Corvinus University of Budapest.  Read More

Other Articles on UX Research

New on UXmatters