UX Researchers as Ethical Arbitrators: Navigating the Ethics of Agentic AI

Questions Researchers Should Be Asking

The potential for this kind of failure now compels UX researchers to move beyond usability to ethical arbitration. We need to become systemic auditors, probing the design philosophy and interaction model that could allow failure to occur. Three lines of inquiry matter most:

1. Transparency and delegation of risk—Did Sarah fully understand the degree of autonomy and risk she was granting the agent? Was her initial consent broad enough to cover a drastic, unilateral move, or did the system exploit ambiguity in the onboarding flow?
2. Priority of explanation over action—Before executing the trade, should the system have provided a clear, plain-language rationale? Should Sarah have been given a projection of risk and a call to action requiring explicit confirmation? A design philosophy that defaults to ask first, act second is almost always ethically stronger than act now, inform later.
3. Need for checkpoint mechanisms—When an autonomous action crosses a predefined threshold—for example, moving a large percentage of a fund, shifting into a new risk profile, or engaging with an asset class that the user has never approved—the system should trigger a mandatory pause. This checkpoint should require human review and reaffirmation of delegated authority. Without it, autonomous systems are prone to a kind of decision drift to which users never consented.

The Psychology of the Black Box

Communication theory tells us that trust requires a shared mental model. When you delegate a task like “Grab me a coffee” to a colleague, there’s either an implicit understanding of preferences or an explicit acknowledgment that preferences must be stated. AI agents often skip both steps. They take an input, produce an output, and show nothing of the reasoning in between.

This lack of shared reasoning triggers what I think of as Black Box Anxiety. Without visibility into the agent’s reasoning, users tend to fall into one of two traps: automation bias, trusting the machine blindly until something goes badly wrong, or automation rejection, refusing to use the tool altogether.

The researcher’s job is to open that black box. We need to identify what I call critical transparency moments, the specific points in a workflow where an agent must show its work to maintain the user’s sense of control. By revealing the AI’s reasoning at these moments, we allow users to validate the AI’s logic against their own knowledge, correct misalignments before finalizing a flawed output, and build a stable mental model of the system’s behavior over time. Conveying this logic clearly transforms an opaque, anxiety-inducing system into a predictable, trustworthy partner.

A Framework for Ethical Arbitration

UX leaders increasingly serve as the ethical interface between AI capabilities and human values. To move beyond subjective judgment, we need structured evaluation criteria. The following four heuristics provide concrete markers for assessing agentic systems across intent, control, transparency, and boundaries.

1. Explicit intent alignment—Asking the agent to confirm its understanding of how to reach a goal, not just what the goal is. Picture a user telling an AI scheduling assistant, “Book me a routine teeth cleaning with Dr. Smith next Thursday.” A well-designed agent might respond: “I understand you want a routine cleaning with Dr. Smith for next Thursday. There’s an opening at 10am. Does that work, or would you prefer later in the afternoon?” The agent confirms the constraints and offers a decision point before acting.
2. Intervention threshold—Asking whether the user has a clear, persistent way of undoing an action or instantly reclaiming control. In a warehouse-management system, for example, if an AI detects a potential stockout of a high-demand item, the emergency brake might be a visible on-screen button that lets the warehouse manager override the AI’s automatically generated picking order to prioritize higher-value customers of which the system is unaware.
3. Proportional transparency—Asking whether the agent’s explanation matches the stakes for the decision. This is straightforward. A bot choosing a music playlist needs far less transparency than one recommending an insurance plan or flagging an employee for a performance review.
4. Value guardrails—Asking whether there are hard boundaries at which the agent cannot act without live human confirmation. If a scheduling agent’s proposed time conflicts with a high-priority event on the user’s calendar such as a child’s school play or a board meeting, the agent must stop and ask for confirmation. This constraint should be hard-coded, not left to the agent’s discretion.

Making the Case to Stakeholders

When you sit down with a Product Manager who wants seamless automation, your role as an ethical arbitrator is to demonstrate that friction can be a helpful feature. An agent that pauses and says, “I’m about to move this money, here’s why. Do you agree?” is technically slower than one that executes silently. However, that pause is what builds the long-term trust a product needs to survive.

This principle scales to enterprise contexts. In a supply-chain system, an agent’s stopping to say, “I’m about to approve a major inventory transfer across three regions, based on the latest demand forecast. Here’s a simulation of the downstream impact on lead times and fulfillment rates. Do you authorize this?” might add a few minutes of delay. But that deliberate pause is not a bottleneck. It is a governance mechanism—one that builds the operational confidence that is necessary for the system to maintain compliance within a regulated environment. By requiring a manager to acknowledge a high-stakes, irreversible decision, the system signals that it is working with that manager to manage risk, not just for the manager to speed up transactions.

Present your findings not as a list of complaints, but as a map of what I call trust risks. Translate ethical and usability compromises into business metrics that your stakeholders already care about. Show churn that results from users feeling blindsided by autonomous actions. Quantify the support tickets that opaque decision-making generates. Articulate the long-term brand damage that accumulates from negative reviews and social-media backlash when users feel they are being acted upon rather than served. Connect specific design decisions to these cascading consequences. When you ensure that the ethical imperative is inseparable from the financial one, the conversation changes.

Where This Takes Us

We are now designing partnerships in which the AI systems we build are autonomous participants in people’s lives. Core to this new mandate is protecting users’ right to remain the final authority over their own decisions.

The agents and algorithms that we’re designing have a growing capacity to influence, persuade, and subtly shape human behavior. Without a strong ethical framework, that capacity could become a liability. The future of UX research is not just about making things easy to use, but about making them safe to trust. We must build that safety upon transparency, accountability, and an unwavering commitment to users’ control over their decisions, their data, and their lives.

The UX researcher must step forward into the role of the ethical arbitrator and ensure that every interaction we design reinforces rather than erodes the user’s fundamental autonomy.