mlsanityData to 7 Oct 2026

Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration

arXiv 2409.12809 ↗First posted 19 Sep 2024cs.HC cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Across various applications, humans increasingly use black-box artificial intelligence (AI) systems without insight into these systems' reasoning. To counter this opacity, explainable AI (XAI) methods promise enhanced transparency and interpretability. While recent studies have explored how XAI affects human-AI collaboration, few have examined the potential pitfalls caused by incorrect explanations. The implications for humans can be far-reaching but have not been explored extensively. To investigate this, we ran a study (n=160) on AI-assisted decision-making in which humans were supported by XAI. Our findings reveal a misinformation effect when incorrect explanations accompany correct AI advice with implications post-collaboration. This effect causes humans to infer flawed reasoning strategies, hindering task execution and demonstrating impaired procedural knowledge. Additionally, incorrect explanations compromise human-AI team-performance during collaboration. With our work, we contribute to HCI by providing empirical evidence for the negative consequences of incorrect explanations on humans post-collaboration and outlining guidelines for designers of AI.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 59 of its 145 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

59 of 145 references.

References · 59

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  5. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  6. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  7. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  8. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  9. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  10. 2019Counterfactual Visual Explanationscited by 312
  11. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  12. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
38 more
  1. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  2. 2020Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthycited by 182
  3. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  4. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 150
  5. 2017Fairness in Machine Learning: Lessons from Political Philosophycited by 148
  6. 2020Making deep neural networks right for the right scientific reasons by interacting with their explanationscited by 141
  7. 2021Expanding Explainability: Towards Social Transparency in AI systemscited by 124
  8. 2023Generative AIcited by 119
  9. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  10. 2019"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanationscited by 111
  11. 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
  12. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  13. 2024Are self-explanations from Large Language Models faithful?cited by 101
  14. 2019Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptionscited by 81
  15. 2023Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanationscited by 80
  16. 2023Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanationscited by 73
  17. 2022"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interactioncited by 63
  18. 2020Does Explainable Artificial Intelligence Improve Human Decision-Making?cited by 49
  19. 2019How model accuracy and explanation fidelity influence user trustcited by 44
  20. 2017Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanationscited by 37
  21. 2018Generating Counterfactual Explanations with Natural Languagecited by 37
  22. 2021Explainability Pitfalls: Beyond Dark Patterns in Explainable AIcited by 36
  23. 2024Complementarity in Human-AI Collaboration: Concept, Sources, and Evidencecited by 35
  24. 2022Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Makingcited by 33
  25. 2023Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfactioncited by 32
  26. 2024On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectivescited by 28
  27. 2022Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Makingcited by 25
  28. 2023The Impact of Imperfect XAI on Human-AI Decision-Makingcited by 21
  29. 2023Effective Human-AI Teams via Learned Natural Language Rules and Onboardingcited by 20
  30. 2024LLMs for XAI: Future Directions for Explaining Explanationscited by 19
  31. 2022On the Influence of Explainable AI on Automation Biascited by 18
  32. 2024ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?cited by 17
  33. 2024Leveraging Large Language Models for Relevance Judgments in Legal Case Retrievalcited by 16
  34. 2023Impact Of Explainable AI On Cognitive Load: Insights From An Empirical Studycited by 15
  35. 2020The Clever Hans Effect in Anomaly Detectioncited by 13
  36. 2022Trust in Human-AI Interaction: Scoping Out Models, Measures, and Methodscited by 13
  37. 2024Explainable AI improves task performance in human-AI collaborationcited by 11
  38. 2021Interpretable Machine Learning: Moving From Mythos to Diagnosticscited by 9

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Towards Human-Understandable Multi-Dimensional Concept Discoverycited by 4
  2. 2025Bias in the Loop: How Humans Evaluate AI-Generated Suggestionscited by 4
  3. 2025From Consumption to Collaboration: Measuring Interaction Patterns to Augment Human Cognition in Open-Ended Taskscited by 3
  4. 2026Cognitive Agency Surrender: Defending Epistemic Sovereignty via Scaffolded AI Frictioncited by 2
  5. 2026Comparables XAI: Faithful Example-based AI Explanations with Counterfactual Trace Adjustmentscited by 1
  6. 2025Development of Mental Models in Human-AI Collaboration: A Conceptual Frameworkcited by 0
  7. 2026Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understandingcited by 0
  8. 2026Supporting Calibrated Reliance in Human-AI Collaboration: Different Strategies for Different Taskscited by 0