mlsanityData to 7 Oct 2026

To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

arXiv 2102.09692 ↗First posted 19 Feb 2021cs.HC cs.AIcited by 118

Not on the map. Why?

Abstract (arXiv)

People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI's suggestion even when that suggestion is wrong. Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it. Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions. Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations. We conducted an experiment (N=199), in which we compared our three cognitive forcing designs to two simple explainable AI approaches and to a no-AI baseline. The results demonstrate that cognitive forcing significantly reduced overreliance compared to the simple explainable AI approaches. However, there was a trade-off: people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most. To audit our work for intervention-generated inequalities, we investigated whether our interventions benefited equally people with different levels of Need for Cognition (i.e., motivation to engage in effortful mental activities). Our results show that, on average, cognitive forcing interventions benefited participants higher in Need for Cognition more. Our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 9 of its 68 references that are in the map. First posted on arXiv 19 Feb 2021.

Builds on

9 of 68 references.

References · 9

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  3. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
  4. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  5. 2018To Trust Or Not To Trust A Classifiercited by 205
  6. 2020Evaluating and Aggregating Feature-based Model Explanationscited by 185
  7. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  8. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
  9. 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 11; 2023: 17; 2024: 35; 2025: 25; 2026: 29.

118 papers.

Sort:

By area:

  1. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  2. 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
  3. 2024What Large Language Models Know and What People Think They Knowcited by 109
  4. 2024When combinations of humans and AI are useful: A systematic review and meta-analysiscited by 100
  5. 2023Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanationscited by 80
  6. 2024Building Machines that Learn and Think with Peoplecited by 73
  7. 2023Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Makingcited by 67
  8. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  9. 2022Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderationcited by 66
  10. 2024AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuancescited by 59
  11. 2024Design Principles for Generative AI Applicationscited by 57
  12. 2024Determinants of LLM-assisted Decision-Makingcited by 55
38 more
  1. 2025Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistantcited by 54
  2. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  3. 2022The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanationscited by 41
  4. 2023Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systemscited by 40
  5. 2024IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedbackcited by 40
  6. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  7. 2024Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Makingcited by 38
  8. 2022Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Supportcited by 37
  9. 2023Improving Human-AI Collaboration With Descriptions of AI Behaviorcited by 35
  10. 2024Trust in AI: Progress, Challenges, and Future Directionscited by 35
  11. 2023CoQuest: Exploring Research Question Co-Creation with an LLM-based Agentcited by 34
  12. 2022Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theorycited by 32
  13. 2022Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imagingcited by 32
  14. 2023Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfactioncited by 32
  15. 2023ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Eventscited by 31
  16. 2021Explainable Medical Imaging AI Needs Human-Centered Design: Guidelines and Evidence from a Systematic Reviewcited by 30
  17. 2024A Decision Theoretic Framework for Measuring AI Reliancecited by 26
  18. 2025Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistantcited by 26
  19. 2022Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Makingcited by 25
  20. 2023Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Makingcited by 25
  21. 2023A Systematic Review on Fostering Appropriate Trust in Human-AI Interactioncited by 21
  22. 2023Human-AI collaboration is not very collaborative yet: A taxonomy of interaction patterns in AI-assisted decision making from a systematic reviewcited by 18
  23. 2025As Confidence Aligns: Exploring the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Makingcited by 18
  24. 2023Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressurecited by 17
  25. 2024Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learningcited by 16
  26. 2024Perceptions of Sentient AI and Other Digital Minds: Evidence from the AI, Morality, and Sentience (AIMS) Surveycited by 15
  27. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  28. 2022Assessing the communication gap between AI models and healthcare professionals: explainability, utility and trust in AI-driven clinical decision-makingcited by 14
  29. 2024Does More Advice Help? The Effects of Second Opinions in AI-Assisted Decision Makingcited by 14
  30. 2024ExpressEdit: Video Editing with Natural Language and Sketchingcited by 14
  31. 2022The Conflict Between Explainable and Accountable Decision-Making Algorithmscited by 13
  32. 2024One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generationscited by 13
  33. 2025Which Contributions Deserve Credit? Perceptions of Attribution in Human-AI Co-Creationcited by 13
  34. 2025Artificial Intelligence Should Genuinely Support Clinical Reasoning and Decision Making To Bridge the Translational Gapcited by 13
  35. 2025AI, Help Me Thinkunicodex2014but for Myself: Assisting People in Complex Decision-Making by Providing Different Kinds of Cognitive Supportcited by 12
  36. 2023Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Makingcited by 11
  37. 2024Contrastive Explanations That Anticipate Human Misconceptions Can Improve Human Decision-Making Skillscited by 11
  38. 2024AI on My Shoulder: Supporting Emotional Labor in Front-Office Roles with an LLM-based Empathetic Coworkercited by 11

50 most cited of 118 All 118 (JSON)