mlsanityData to 14 Oct 2026

Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting

arXiv 2410.12284 ↗First posted 16 Oct 2024cs.HC cs.CL cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

The growing capabilities of AI models are leading to their wider use, including in safety-critical domains. Explainable AI (XAI) aims to make these models safer to use by making their inference process more transparent. However, current explainability methods are seldom evaluated in the way they are intended to be used: by real-world end users. To address this, we conducted a large-scale user study with 85 healthcare practitioners in the context of human-AI collaborative chest X-ray analysis. We evaluated three types of explanations: visual explanations (saliency maps), natural language explanations, and a combination of both modalities. We specifically examined how different explanation types influence users depending on whether the AI advice and explanations are factually correct. We find that text-based explanations lead to significant over-reliance, which is alleviated by combining them with saliency maps. We also observe that the quality of explanations, that is, how much factually correct information they entail, and how much this aligns with AI correctness, significantly impacts the usefulness of the different explanation types.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 39 of its 74 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

39 of 74 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  5. 2018Sanity Checks for Saliency Mapscited by 1,156
  6. 2020Concept Bottleneck Modelscited by 1,032
  7. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  8. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  9. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  10. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  11. 2020Measuring Association Between Labels and Free-Text Rationalescited by 160
  12. 2016Generating Visual Explanationscited by 158
27 more
  1. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  2. 2020Debugging Tests for Model Explanationscited by 147
  3. 2019What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Usecited by 135
  4. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  5. 2021Do Concept Bottleneck Models Learn as Intended?cited by 116
  6. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  7. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  8. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  9. 2019Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reportscited by 96
  10. 2020Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?cited by 95
  11. 2020Human-centered Explainable AI: Towards a Reflective Sociotechnical Approachcited by 90
  12. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  13. 2021e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Taskscited by 84
  14. 2024A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretationcited by 81
  15. 2021This Looks Like That... Does it? Shortcomings of Latent Space Prototype Interpretability in Deep Networkscited by 65
  16. 2022Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AIcited by 40
  17. 2020How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labelscited by 39
  18. 2020Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Predictioncited by 36
  19. 2022Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imagingcited by 32
  20. 2019Evaluating Explanation Without Ground Truth in Interpretable Machine Learningcited by 29
  21. 2024The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Modelscited by 29
  22. 2022Explaining Chest X-ray Pathologies in Natural Languagecited by 27
  23. 2023The Impact of Imperfect XAI on Human-AI Decision-Makingcited by 21
  24. 2018Analyzing Neuroimaging Data Through Recurrent Deep Learning Modelscited by 15
  25. 2018Producing radiologist-quality reports for interpretable artificial intelligencecited by 11
  26. 2018ClinicalVis: Supporting Clinical Task-Focused Design Evaluationcited by 4
  27. 2024On Biases in a UK Biobank-based Retinal Image Classification Modelcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025A Necessary Step toward Faithfulness: Measuring and Improving Consistency in Free-Text Explanationscited by 3
  2. 2025A Taxonomy for Design and Evaluation of Prompt-Based Natural Language Explanationscited by 1
  3. 2025AI Chaperones Are (Really) All You Need to Prevent Parasocial Relationships with Chatbotscited by 1
  4. 2025NeuroFaith: Evaluating LLM Self-Explanation Faithfulness via Internal Representation Alignmentcited by 0
  5. 2025Explainable Melanoma Diagnosis with Contrastive Learning and LLM-based Report Generationcited by 0