mlsanityData to 7 Oct 2026

How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps

arXiv 2404.16042 ↗First posted 3 Apr 2024cs.HC cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Saliency maps can explain how deep neural networks classify images. But are they actually useful for humans? The present systematic review of 68 user studies found that while saliency maps can enhance human performance, null effects or even costs are quite common. To investigate what modulates these effects, the empirical outcomes were organised along several factors related to the human tasks, AI performance, XAI methods, images to be classified, human participants and comparison conditions. In image-focused tasks, benefits were less common than in AI-focused tasks, but the effects depended on the specific cognitive requirements. Moreover, benefits were usually restricted to incorrect AI predictions in AI-focused tasks but to correct ones in image-focused tasks. XAI-related factors had surprisingly little impact. The evidence was limited for image- and human-related factors and the effects were highly dependent on the comparison conditions. These findings may support the design of future user studies.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 48 of its 88 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

48 of 88 references.

References · 48

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  5. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  6. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 177
  7. 2020Debugging Tests for Model Explanationscited by 146
  8. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  9. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
  10. 2020Reliable Post hoc Explanations: Modeling Uncertainty in Explainabilitycited by 125
  11. 2019XRAI: Better Attributions Through Regionscited by 116
  12. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
36 more
  1. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  2. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 87
  3. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  4. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 84
  5. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  6. 2018Do Explanations make VQA Models more Predictable to a Human?cited by 51
  7. 2019Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attributioncited by 50
  8. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  9. 2020How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labelscited by 39
  10. 2020Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Predictioncited by 36
  11. 2020One Explanation is Not Enough: Structured Attention Graphs for Image Classificationcited by 32
  12. 2018Teaching Categories to Human Learners with Visual Explanationscited by 30
  13. 2022A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Makingcited by 29
  14. 2022Explainable Artificial Intelligence (XAI) from a user perspective- A synthesis of prior literature and problematizing avenues for future researchcited by 20
  15. 2020A Study on Multimodal and Interactive Explanations for Visual Question Answeringcited by 14
  16. 2017Embedding Deep Networks into Visual Explanationscited by 12
  17. 2021Explainable Artificial Intelligence for Human Decision-Support System in Medical Domaincited by 12
  18. 2023ODAM: Gradient-based instance-specific visual explanations for object detectioncited by 12
  19. 2020Debiased-CAM to mitigate image perturbations with faithful visual explanations of machine learningcited by 11
  20. 2021Mitigating belief projection in explainable artificial intelligence via Bayesian Teachingcited by 10
  21. 2022Trust and Reliance in XAI -- Distinguishing Between Attitudinal and Behavioral Measurescited by 10
  22. 2022A Psychological Theory of Explainabilitycited by 10
  23. 2020Measure Utility, Gain Trust: Practical Advice for XAI Researchercited by 9
  24. 2020Explaining Local, Global, And Higher-Order Interactions In Deep Learningcited by 8
  25. 2021Crowdsourcing Evaluation of Saliency-based XAI Methodscited by 7
  26. 2022Perception Visualization: Seeing Through the Eyes of a DNNcited by 5
  27. 2023Modeling Reliance on XAI Indicating Its Purpose and Attentioncited by 4
  28. 2019A psychophysics approach for quantitative comparison of interpretable computer vision modelscited by 3
  29. 2023Designing a Direct Feedback Loop between Humans and Convolutional Neural Networks through Local Explanationscited by 3
  30. 2023Interpretability is in the eye of the beholder: Human versus artificial classification of image segments generated by humans versus XAIcited by 3
  31. 2021Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Modelscited by 2
  32. 2021Canonical Saliency Maps: Decoding Deep Face Modelscited by 2
  33. 2021Explainable AI for Natural Adversarial Imagescited by 2
  34. 2022Responsibility: An Example-based Explainable AI approach via Training Process Inspectioncited by 2
  35. 2019Minimalistic Explanations: Capturing the Essence of Decisionscited by 1
  36. 2023The benefits and costs of explainable artificial intelligence in visual quality control: Evidence from fault detection performance and eye movementscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  2. 2025DocVXQA: Context-Aware Visual Explanations for Document Question Answeringcited by 5
  3. 2025Are explainable AI (XAI) evaluation strategies aligned? Comparing subjective, objective, and mathematical evaluation measures using saliency mapscited by 4
  4. 2025WebXAII: an open-source web framework to study human-XAI interactioncited by 0
  5. 2026Associations between echocardiographic traits and AI-ECG predictions of heart failurecited by 0
  6. 2026Federated Explainable Artificial Intelligence: Roles, Architectures, Evaluation, and Open Challengescited by 0