mlsanityData to 7 Oct 2026

Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies

arXiv 2502.08554 ↗First posted 12 Feb 2025cs.HC cs.AIcited by 39

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) can produce erroneous responses that sound fluent and convincing, raising the risk that users will rely on these responses as if they were correct. Mitigating such overreliance is a key challenge. Through a think-aloud study in which participants use an LLM-infused application to answer objective questions, we identify several features of LLM responses that shape users' reliance: explanations (supporting details for answers), inconsistencies in explanations, and sources. Through a large-scale, pre-registered, controlled experiment (N=308), we isolate and study the effects of these features on users' reliance, accuracy, and other measures. We find that the presence of explanations increases reliance on both correct and incorrect responses. However, we observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies. We discuss the implications of these findings for fostering appropriate reliance on LLMs.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 66 of its 138 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

66 of 138 references.

References · 66

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  11. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
38 more
  1. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  2. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  3. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  4. 2019Defending Against Neural Fake Newscited by 553
  5. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  6. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  7. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  8. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  9. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  10. 2023Explainability for Large Language Models: A Surveycited by 331
  11. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  12. 2023Faithful Chain-of-Thought Reasoningcited by 285
  13. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  14. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  15. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  16. 2022Talking About Large Language Modelscited by 174
  17. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  18. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  19. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
  20. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  21. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 124
  22. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  23. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  24. 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
  25. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  26. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  27. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  28. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  29. 2020"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humanscited by 86
  30. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  31. 2023Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanationscited by 73
  32. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
  33. 2023Evaluating Language Models for Mathematics through Interactionscited by 69
  34. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  35. 2022"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interactioncited by 63
  36. 2021Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Makingcited by 58
  37. 2022Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learningcited by 53
  38. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 28.

39 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Consistency in Language Models: Current Landscape, Challenges, and Future Directionscited by 14
  3. 2025Measuring and mitigating overreliance to build human-compatible AIcited by 9
  4. 2026AI Assistance Reduces Persistence and Hurts Independent Performancecited by 7
  5. 2025Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translationcited by 5
  6. 2025Humans overrely on overconfident language models, across languagescited by 3
  7. 2026Reflexis: Supporting Reflexivity and Rigor in Collaborative Qualitative Analysis through Design for Deliberationcited by 3
  8. 2025Let's Get You Hired: A Job Seeker's Perspective on Multi-Agent Recruitment Systems for Explaining Hiring Decisionscited by 2
  9. 2025Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewardscited by 2
  10. 2026PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-based Scholarly Q&Acited by 2
  11. 2026More Than "Means to an End": Supporting Reasoning with Transparently Designed AI Data Science Processescited by 2
  12. 2026Position: Explainability Research Must Prioritize Foundations over Ad-hoc Methodscited by 2
27 more
  1. 2025Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Workcited by 1
  2. 2026From Junior to Senior: Allocating Agency and Navigating Professional Growth in Agentic AI-Mediated Software Engineeringcited by 1
  3. 2026What is Safety? Corporate Discourse, Power, and the Politics of Generative AI Safetycited by 1
  4. 2026From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistantscited by 1
  5. 2026Towards an Appropriate Level of Reliance on AI: A Preliminary Reliance-Control Framework for AI in Software Engineeringcited by 1
  6. 2026Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaborationcited by 1
  7. 2025Beyond Autocomplete: Designing CopilotLens Towards Transparent and Explainable AI Coding Agentscited by 0
  8. 2025Does In-IDE Calibration of Large Language Models work at Scale?cited by 0
  9. 2025Decision-Making Amid Information-Based Threats in Sociotechnical Systems: A Reviewcited by 0
  10. 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Makingcited by 0
  11. 2026From Toil to Thought: Designing for Strategic Exploration and Responsible AI in Systematic Literature Reviewscited by 0
  12. 2026Human Thinking under Plural LLM Assistance: Mathematical Problem Solving and Open-Ended Writingcited by 0
  13. 2026Justified or Just Convincing? Error Verifiability as a Dimension of LLM Qualitycited by 0
  14. 2026U-Define: Designing User Workflows for Hard and Soft Constraints in LLM-Based Planningcited by 0
  15. 2026Toward Human-AI Complementarity Across Diverse Taskscited by 0
  16. 2026PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AIcited by 0
  17. 2026How do Humans Process AI-generated Hallucination Contents: a Neuroimaging Studycited by 0
  18. 2026RuleEdit: Failure-Guided Human-AI Model Editing with Prospective Impact Previewcited by 0
  19. 2026Explain Like I'm 5 or Whatever I Choose: Evaluating the Interactive Potential of Language Model Responsescited by 0
  20. 2026Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Promptscited by 0
  21. 2026AI Assistance for Human Review of Default Judgmentscited by 0
  22. 2026Security and Privacy in Agentic AI: Grand Challenges and Future Directionscited by 0
  23. 2026The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systemscited by 0
  24. 2026Beyond "ChatGPT Can Make Mistakes": Designing Interventions to Support Metacognitive Monitoring in AI-Assisted Workcited by 0
  25. 2026Annie, Are You Okay? How Style- and Context-Based Personalization Shape AI-Assisted Decision-Makingcited by 0
  26. 2026Signals of AI Hallucination: Designing Hallucination-Aware Cues for Embodied Conversational Agents in VRcited by 0
  27. 2026Cybernetic and Epistemic: A Missing Vocabulary for Trustworthy Agentic Delegationcited by 0