Designing for Doubt: How to Prevent Automation Complacency in AI Workflows

What Automation Complacency Is and Isn’t

Complacency is not laziness. It occurs even in highly trained, deeply motivated professionals. The Air France pilots were experienced and highly trained. Radar operators who miss targets after spending thirty minutes on watch are both attentive and dedicated to their work, but are prone to trusting automation. Enterprise users who stop checking AI-generated reports after a month were diligent reviewers in week one. Complacency is a predictable cognitive response to reliable automation.

Complacency is about judgment. Users stop applying their own assessments because they have learned to trust a system’s assessments instead. For example, consider the case of software engineers who are reviewing AI-generated code. They might see a security-fix recommendation and click Accept because the AI has a 99%-correct track record but fail to notice that the fix would simultaneously introduce a new performance bottleneck. They are fully engaged, but they have deferred their professional judgment to the machine. People can be fully attentive and still complacent.

Complacency is distinct from trust—although trust is its prerequisite. Trust is an attitude: a positive feeling and belief that a system is reliable. Complacency is a behavior: the act of diminishing one’s own verifications because of the belief that the system doesn’t require them. Trust is appropriate when we calibrate it to a system’s actual reliability. Complacency is what happens when trust becomes unquestioning.

Parasuraman and Manzey, in their 2010 framework, identified three conditions that produce complacency:

1. Automation is highly reliable.
2. The operator has a high workload with competing tasks.
3. The operator has had sustained positive experience with the system.

All three of these conditions are present in virtually every enterprise AI deployment, and the presence of all three conditions sets the stage for the next phase of the user’s predictable response. I’ll detail this mechanism in the next section.

The Reliability Trap

The reliability trap is a gradual process that users enter into over time. In the first weeks of using an AI tool, users check its outputs against their own judgment. They override recommendations with which they disagree. They verify the AI’s calculations. They treat the AI’s outputs as suggestions to evaluate, not conclusions to accept.

As the AI proves its reliability over dozens, then hundreds of interactions, Users’ checking behavior shifts. They stop verifying things the AI has always gotten right. They check selectively, focusing on areas where they’ve seen errors before and skipping areas where they haven’t. This is rational: allocating scarce attention to the highest-risk areas.

But this shift doesn't stop at rational selectivity. Over weeks and months, users’ baseline assumption changes. It moves from “the AI’s output is probably right, but I should verify it” to the assumption that “the AI’s output is right unless something looks obviously wrong.” Then, subtly, the assumption becomes “the AI’s output is right.” This final state is complacency. Users are no longer evaluating the AI’s recommendations. They are simply accepting them. They’ve functionally disabled their own professional judgment—the judgment that was supposed to serve as the safety check—because of the system’s track record of being correct.

I’ve watched this progression unfold in domains that include automated legal review and fraud detection. The timeline varies for each individual and according to the task complexity, but the trajectory is consistent. By the third month, most users have settled into a pattern where the AI’s recommendations have become the default, and they activate their own judgment only when something triggers a feeling that something is off. When I ask users to articulate what would trigger that feeling, they struggle. The threshold for users’ engaging their own judgment has risen so high that, for routine decisions, they perceive that the cognitive cost of overriding the AI is higher than the risk of being wrong. This is the danger of the reliability trap: users disable their safety checks without even noticing.

Automation Bias: Complacency’s More Dangerous Cousin

There is a more aggressive variant of complacency called automation bias: the tendency for users to follow automated systems’ recommendations even when independent evidence contradicts them. While complacency is passive, causing the user not to check, automation bias is active—the user checks and identifies a discrepancy but defers to the machine anyway.

Automation bias is well documented in medical decision-support systems. Clinicians who receive AI-generated diagnostic suggestions sometimes follow those suggestions even when their own clinical assessment has pointed in a different direction. The AI’s confidence overrode their own.

In my enterprise research, I’ve observed a subtler version of this phenomenon. Users who encounter an AI recommendation that doesn’t match their intuition often adjust their own thinking rather than override the AI. One participant described the experience as follows: “I thought the pricing looked high, but the system seemed confident, so I figured maybe I was wrong.” The AI didn't force the user to accept its recommendation. The user voluntarily suppressed his own judgment in favor of the machine’s.

This is automation bias operating at the level of self-doubt. The user’s expertise tells him something is off. But the AI’s confidence tells him to trust the system. In the moment of tension between the two, the AI wins—not because it’s right, but because it’s more certain.

Four Design Patterns for Appropriate Trust

While outright distrust of an AI can lead to costly verifications and rework, users’ unquestioning complacency leads to catastrophic errors. Our UX design goal is to gain users’ calibrated trust, where their reliance on a system matches the system’s actual reliability, including its failure modes. Let’s look at four design patterns that let us better achieve this goal.

Pattern 1: Build in friction by design.

Require the user to engage with the AI’s reasoning before accepting its output. This could be as simple as showing a one-sentence summary of the AI’s logic and asking the user to confirm one specific aspect of it—for example, “This price is based on the Q2 discount schedule. Does that apply to this customer?” There is minimal friction for the user, the cognitive engagement is meaningful, and the question is specific enough to activate the user’s domain knowledge rather than prompting a reflexive yes.

Alternatively, in an AI-assisted compliance-review tool, the system might flag a contract clause such as “Low Risk” and require the user select a check box to affirm that he has verified the definition of a specific legal term—for example, material adverse change—as it applies to the current jurisdiction. This is another way to force active confirmation of a crucial detail that underlies the AI’s conclusion, without demanding a full manual review.

Pattern 2: Calibrate the AI’s confidence signals.

When the AI presents each and every recommendation with the same apparent certainty, users learn to treat all outputs as equally reliable. This trains complacency. Instead, vary the AI’s expressed confidence. Users can approve high-confidence outputs with minimal friction, while low-confidence outputs should require more engagement. This teaches users that some outputs deserve more scrutiny than others, which is the opposite of the blanket acceptance that complacency produces.

For example, a machine-learning system that reviews incoming customer-support tickets might show a “98% confidence: Personally Identifiable Information (PII) detected” badge on highly certain flags, allowing immediate automated redaction. If the system’s confidence score drops to “65% confidence: PII possible,” it routes the ticket directly to a human reviewer, who then must click a verification button, “I confirm that PII is present,” before proceeding. This difference in presentation and the required action prevents the reviewer from developing a habit of reflexively trusting every flag.

Pattern 3: Maintain users’ independent judgment by periodically requiring unassisted tasks.

If users never exercise their independent judgment, their ability to judge AI outputs erodes, deepening their complacency. We must design for skill preservation. One way to achieve this is to build required, periodic “manual flying” moments into a workflow.

For an AI-assisted tool that generates complex quarterly reports, this might mean that every tenth report or a randomized set of high-stakes reports requires the user to toggle the AI off and generate a key component of the report manually from the raw data. This forces users to actively re-engage their domain knowledge, recalibrating their baseline judgment before they return to overseeing the automated process. Think of this as the cognitive equivalent of a pilot’s mandatory manual-flying requirements.

For an AI-coding assistant that automates 95% of routine infrastructure-as-code (IaC) configuration, a realistic pattern would force a software developer to manually write the configuration file for a new deployment every fifth sprint or require the developer to use a legacy, non-AI tool to troubleshoot a high-priority bug once a month. This would ensure that the developer’s core syntax and troubleshooting skills remain current, so he is capable of intervening when the AI misjudges or fails on a novel edge case.

Pattern 4: Measure complacency indicators and treat them as system-health metrics.

Track override rates, time to acceptance, and the correlation between AI confidence and user acceptance. In a healthy system, users should accept high-confidence outputs more readily than low-confidence outputs. If they accept both at the same rate, complacency has set in. Surface these metrics to the design team, not as individual performance data, but as signals that the oversight design needs adjustment.

The Space Between Trust and Acceptance

Trust in AI is valuable. It means the user has learned that the system works, that its recommendations are generally sound, and that delegating to it is a productive use of the user’s time. This is the outcome we want.

Complacency happens when trust closes the gap between “I believe this system works” and “I no longer evaluate whether it has worked this time.” This is the gap where human judgment lives. It is where users catch errors, flag edge cases, and discover the AI’s limitations. Keeping this gap open requires good design, because the natural trajectory of human cognition in the presence of reliable automation is to close it.

The organizations that get this right will create AI tools that people trust, but verify. The organizations that do not will create AI tools that are trusted but remain unexamined. The difference between the two becomes visible only when something goes wrong. We can avoid such catastrophic moments by choosing to design for doubt. 

References and Further Reading

Aviation Safety Network. “Loss of Control Accident Airbus A330-203 F, Monday 1 June 2009.” Aviation Safety Network, March 19, 2025. Retrieved May 18, 2026.

Jason Blair. “The Dangers of Overreliance on Automation.” FAA Safety Briefing Magazine, May 2, 2025. Retrieved May 18, 2026.

Lydia Harbarth, Eva Gößwein, Daniel Bodemer, and Lenka Schnaubert. “Over-Trusting AI Recommendations: How System and Person Variables Affect Dimensions of Complacency.” International Journal of Human–Computer Interaction, 2024. Retrieved May 18, 2026.

Peng Liu. “Reflections on Automation Complacency.” International Journal of Human–Computer Interaction, 2024. Retrieved May 27, 2026.

Stephanie M. Merritt, Alicia Ako-Brew, W. J. Bryant, A. Staley, M. McKenna, A. Leone, and Lei Shirase. “Automation-Induced Complacency Potential: Development and Validation of a New Scale.” Frontiers in Psychology, 2019. Retrieved May 20, 2026.

Raja Parasuraman and Dietrich H. Manzey. “Complacency and Bias in Human Use of Automation: An Attentional Integration.” Human Factors: The Journal of the Human Factors and Ergonomics Society, 2010. Retrieved May 20, 2026.

Michael I. Saadeh, Joel Janhonen, Emily Beer, Camille Castelyn, and David Hoffman. “Automation Complacency: Risks of Abdicating Medical Decision Making.” AI Ethics, 2025. Retrieved May 20, 2026.

Ryan W. Wohleber, Gerald Matthews, Jinchao Lin, James L. Szalma, Gloria L. Calhoun, Gregory. J. Funke, C.Y. Peter Chiu, and Heath A. Ruff. “Vigilance and Automation Dependence in Operation of Multiple Unmanned Aerial Systems (UAS): A Simulation Study.” Human Factors: The Journal of the Human Factors and Ergonomics Society, 2018. Retrieved May 20, 2026.