# Measuring and mitigating overreliance to build human-compatible AI

arXiv: https://arxiv.org/abs/2509.08010 · HTML: https://mlsanity.com/paper/2509.08010/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 112 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Builds on (most cited first, 29 of 29)

- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations](https://mlsanity.com/paper/2302.02187/) (2023)
- [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023)
- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces](https://mlsanity.com/paper/2309.14459/) (2023)
- [Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making](https://mlsanity.com/paper/2301.05809/) (2023)
- [Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making](https://mlsanity.com/paper/2010.07938/) (2020)
- [Design Principles for Generative AI Applications](https://mlsanity.com/paper/2401.14484/) (2024)
- [Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant](https://mlsanity.com/paper/2502.01390/) (2025)
- [Multicalibration for Confidence Scoring in LLMs](https://mlsanity.com/paper/2404.04689/) (2024)
- [Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems](https://mlsanity.com/paper/2301.11333/) (2023)
- [Cocoa: Co-Planning and Co-Execution with AI Agents](https://mlsanity.com/paper/2412.10999/) (2024)
- [Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies](https://mlsanity.com/paper/2502.08554/) (2025)
- [Understanding Human Intelligence through Human Limitations](https://mlsanity.com/paper/2009.14050/) (2020)
- [Towards interactive evaluations for interaction harms in human-AI systems](https://mlsanity.com/paper/2405.10632/) (2024)
- [A Decision Theoretic Framework for Measuring AI Reliance](https://mlsanity.com/paper/2401.15356/) (2024)
- [Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function](https://mlsanity.com/paper/2406.01382/) (2024)
- [Training language models to be warm and empathetic makes them less reliable and more sycophantic](https://mlsanity.com/paper/2507.21919/) (2025)
- [Modulating Language Model Experiences through Frictions](https://mlsanity.com/paper/2407.12804/) (2024)
- [Warmth and competence in human-agent cooperation](https://mlsanity.com/paper/2201.13448/) (2022)
- [Pitfalls of Evidence-Based AI Policy](https://mlsanity.com/paper/2502.09618/) (2025)
- [Relational Norms for Human-AI Cooperation](https://mlsanity.com/paper/2502.12102/) (2025)
- [Understanding Practices around Computational News Discovery Tools in the Domain of Science Journalism](https://mlsanity.com/paper/2311.06864/) (2023)
- [How Individual Traits and Language Styles Shape Preferences In Open-ended User-LLM Interaction: A Preliminary Study](https://mlsanity.com/paper/2504.17083/) (2025)
- [When Should Algorithms Resign? A Proposal for AI Governance](https://mlsanity.com/paper/2402.18326/) (2024)
- [From Awareness to Action: Exploring End-User Empowerment Interventions for Dark Patterns in UX](https://mlsanity.com/paper/2310.17846/) (2023)
- [FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines](https://mlsanity.com/paper/2307.15475/) (2023)

## Built on by (most cited first, 12 of 12)

- [How are AI agents used? Evidence from 177,000 MCP tools](https://mlsanity.com/paper/2603.23802/) (2026)
- [Critical or Compliant? The Double-Edged Sword of Reasoning in Chain-of-Thought Explanations](https://mlsanity.com/paper/2511.12001/) (2025)
- [Why Human Guidance Matters in Collaborative Vibe Coding](https://mlsanity.com/paper/2602.10473/) (2026)
- [Overseeing Agents Without Constant Oversight: Challenges and Opportunities](https://mlsanity.com/paper/2602.16844/) (2026)
- [Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure](https://mlsanity.com/paper/2605.27996/) (2026)
- [Under the Influence: Quantifying Persuasion and Vigilance in Large Language Models](https://mlsanity.com/paper/2602.21262/) (2026)
- [Receptiveness, Not Sycophancy: Distinguishing Engagement from Deference in Language Models](https://mlsanity.com/paper/2609.26579/) (2026)
- [What Did They Mean? How LLMs Resolve Ambiguous Social Situations across Perspectives and Roles](https://mlsanity.com/paper/2604.23942/) (2026)
- [Justified or Just Convincing? Error Verifiability as a Dimension of LLM Quality](https://mlsanity.com/paper/2604.04418/) (2026)
- [AI Models Can Predict and Collaboratively Modulate Human Memory Search](https://mlsanity.com/paper/2608.26152/) (2026)
- [The Work Behind Delegation: A Framework for Supervising AI Coding Agents](https://mlsanity.com/paper/2609.24234/) (2026)
- [Safety Nudges: User-Facing Interventions for Real-Time AI Risk Awareness](https://mlsanity.com/paper/2609.26865/) (2026)

## Abstract (arXiv)

Large language models (LLMs) distinguish themselves from previous technologies by functioning as collaborative ``thought partners,'' capable of engaging more fluidly in natural language on a range of tasks. As LLMs increasingly influence consequential decisions across diverse domains from healthcare to personal advice, the risk of overreliance -- relying on LLMs beyond their capabilities -- grows. This paper argues that measuring and mitigating overreliance must become central to LLM research and deployment. First, we consolidate risks from overreliance at both the individual and societal levels, including high-stakes errors, governance challenges, and cognitive deskilling. Then, we explore LLM characteristics, system design features, and user cognitive biases that together raise serious and unique concerns about overreliance on LLMs in practice. We also examine historical approaches for measuring overreliance, identifying three important gaps and proposing three promising directions to improve measurement. Finally, we propose mitigation strategies that can be pursued to ensure LLMs augment rather than undermine human capabilities.