mlsanityData to 7 Oct 2026

Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study

arXiv 2002.00772 ↗First posted 3 Feb 2020cs.HCcited by 86

Not on the map. Why?

Abstract (arXiv)

Convolutional neural networks (CNNs) offer great machine learning performance over a range of applications, but their operation is hard to interpret, even for experts. Various explanation algorithms have been proposed to address this issue, yet limited research effort has been reported concerning their user evaluation. In this paper, we report on an online between-group user study designed to evaluate the performance of "saliency maps" - a popular explanation algorithm for image classification applications of CNNs. Our results indicate that saliency maps produced by the LRP algorithm helped participants to learn about some specific image features the system is sensitive to. However, the maps seem to provide very limited help for participants to anticipate the network's output for new images. Drawing on our findings, we highlight implications for design and further research on explainable AI. In particular, we argue the HCI and AI communities should look beyond instance-level explanations.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 16; 2023: 20; 2024: 17; 2025: 9; 2026: 9.

86 papers.

Sort:

By area:

  1. 2021Expanding Explainability: Towards Social Transparency in AI systemscited by 124
  2. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 87
  3. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 85
  4. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 84
  5. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  6. 2021Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanationscited by 53
  7. 2023In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Makingcited by 50
  8. 2021The Who in XAI: How AI Background Shapes Perceptions of AI Explanationscited by 48
  9. 2022Metrics for saliency map evaluation of deep learning explanation methodscited by 40
  10. 2021Explainability Pitfalls: Beyond Dark Patterns in Explainable AIcited by 36
  11. 2023Explainable Deep Reinforcement Learning: State of the Art and Challengescited by 33
  12. 2023Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAIcited by 33
38 more
  1. 2021Question-Driven Design Process for Explainable AI User Experiencescited by 31
  2. 2021Temporal Dependencies in Feature Importance for Time Series Predictionscited by 27
  3. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  4. 2024Explaining Time Series via Contrastive and Locally Sparse Perturbationscited by 22
  5. 2022Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomenacited by 21
  6. 2022Machine Explanations and Human Understandingcited by 20
  7. 2023Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Makingcited by 18
  8. 2021Challenges for machine learning in clinical translation of big data imaging studiescited by 15
  9. 2022On the Relationship Between Explanation and Prediction: A Causal Viewcited by 14
  10. 2022Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Datasetcited by 13
  11. 2023Explainability in AI Policies: A Critical Review of Communications, Reports, Regulations, and Standards in the EU, US, and UKcited by 13
  12. 2022Towards a multi-stakeholder value-based assessment framework for algorithmic systemscited by 12
  13. 2023FINER: Enhancing State-of-the-art Classifiers with Feature Attribution to Facilitate Security Analysiscited by 12
  14. 2023Interpretable Sparsification of Brain Graphs: Better Practices and Effective Designs for Graph Neural Networkscited by 11
  15. 2023ALMANACS: A Simulatability Benchmark for Language Model Explainabilitycited by 11
  16. 2023IRIS: Interpretable Rubric-Informed Segmentation for Action Quality Assessmentcited by 9
  17. 2024Decoding AI's Nudge: A Unified Framework to Predict Human Behavior in AI-assisted Decision Makingcited by 9
  18. 2021Automatic Diagnosis of COVID-19 from CT Images using CycleGAN and Transfer Learningcited by 7
  19. 2021Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutionscited by 7
  20. 2023Error Discovery by Clustering Influence Embeddingscited by 7
  21. 2021Where do Models go Wrong? Parameter-Space Saliency Maps for Explainabilitycited by 6
  22. 2022Monitoring Shortcut Learning using Mutual Informationcited by 6
  23. 2022Assessing Out-of-Domain Language Model Performance from Few Examplescited by 6
  24. 2023Model-agnostic explainable artificial intelligence for object detection in image datacited by 6
  25. 2024How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency mapscited by 6
  26. 2024Identifying Bias in Deep Neural Networks Using Image Transformscited by 6
  27. 2025Evaluating Model Explanations without Ground Truthcited by 6
  28. 2021Evaluation of Saliency-based Explainability Methodcited by 5
  29. 2022Perception Visualization: Seeing Through the Eyes of a DNNcited by 5
  30. 2024Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Settingcited by 5
  31. 2023Predictability and Comprehensibility in Post-Hoc XAI Methods: A User-Centered Analysiscited by 4
  32. 2024Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learningcited by 4
  33. 2025Are explainable AI (XAI) evaluation strategies aligned? Comparing subjective, objective, and mathematical evaluation measures using saliency mapscited by 4
  34. 2022Explainable Predictive Process Monitoring: A User Evaluationcited by 3
  35. 2022A Rigorous Study Of The Deep Taylor Decompositioncited by 3
  36. 2024Explaining Multi-modal Large Language Models by Analyzing their Vision Perceptioncited by 3
  37. 2022Responsibility: An Example-based Explainable AI approach via Training Process Inspectioncited by 2
  38. 2023How good Neural Networks interpretation methods really are? A quantitative benchmarkcited by 2

50 most cited of 86 All 86 (JSON)