# Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study

arXiv: https://arxiv.org/abs/2002.00772 · HTML: https://mlsanity.com/paper/2002.00772/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2020.

## Built on by (most cited first, 30 of 86)

- [Expanding Explainability: Towards Social Transparency in AI systems](https://mlsanity.com/paper/2101.04719/) (2021)
- [The effectiveness of feature attribution methods and its correlation with automatic evaluation scores](https://mlsanity.com/paper/2105.14944/) (2021)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations](https://mlsanity.com/paper/2210.11584/) (2022)
- [Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation](https://mlsanity.com/paper/2212.04629/) (2022)
- [Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanations](https://mlsanity.com/paper/2106.12723/) (2021)
- [In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making](https://mlsanity.com/paper/2305.07722/) (2023)
- [The Who in XAI: How AI Background Shapes Perceptions of AI Explanations](https://mlsanity.com/paper/2107.13509/) (2021)
- [Metrics for saliency map evaluation of deep learning explanation methods](https://mlsanity.com/paper/2201.13291/) (2022)
- [Explainability Pitfalls: Beyond Dark Patterns in Explainable AI](https://mlsanity.com/paper/2109.12480/) (2021)
- [Explainable Deep Reinforcement Learning: State of the Art and Challenges](https://mlsanity.com/paper/2301.09937/) (2023)
- [Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI](https://mlsanity.com/paper/2302.00799/) (2023)
- [Question-Driven Design Process for Explainable AI User Experiences](https://mlsanity.com/paper/2104.03483/) (2021)
- [Temporal Dependencies in Feature Importance for Time Series Predictions](https://mlsanity.com/paper/2107.14317/) (2021)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Explaining Time Series via Contrastive and Locally Sparse Perturbations](https://mlsanity.com/paper/2401.08552/) (2024)
- [Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena](https://mlsanity.com/paper/2206.05487/) (2022)
- [Machine Explanations and Human Understanding](https://mlsanity.com/paper/2202.04092/) (2022)
- [Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making](https://mlsanity.com/paper/2308.04375/) (2023)
- [Challenges for machine learning in clinical translation of big data imaging studies](https://mlsanity.com/paper/2107.05630/) (2021)
- [On the Relationship Between Explanation and Prediction: A Causal View](https://mlsanity.com/paper/2212.06925/) (2022)
- [Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Dataset](https://mlsanity.com/paper/2204.11642/) (2022)
- [Explainability in AI Policies: A Critical Review of Communications, Reports, Regulations, and Standards in the EU, US, and UK](https://mlsanity.com/paper/2304.11218/) (2023)
- [Towards a multi-stakeholder value-based assessment framework for algorithmic systems](https://mlsanity.com/paper/2205.04525/) (2022)
- [FINER: Enhancing State-of-the-art Classifiers with Feature Attribution to Facilitate Security Analysis](https://mlsanity.com/paper/2308.05362/) (2023)
- [Interpretable Sparsification of Brain Graphs: Better Practices and Effective Designs for Graph Neural Networks](https://mlsanity.com/paper/2306.14375/) (2023)
- [ALMANACS: A Simulatability Benchmark for Language Model Explainability](https://mlsanity.com/paper/2312.12747/) (2023)
- [IRIS: Interpretable Rubric-Informed Segmentation for Action Quality Assessment](https://mlsanity.com/paper/2303.09097/) (2023)
- [Decoding AI's Nudge: A Unified Framework to Predict Human Behavior in AI-assisted Decision Making](https://mlsanity.com/paper/2401.05840/) (2024)
- [Error Discovery by Clustering Influence Embeddings](https://mlsanity.com/paper/2312.04712/) (2023)

## Abstract (arXiv)

Convolutional neural networks (CNNs) offer great machine learning performance over a range of applications, but their operation is hard to interpret, even for experts. Various explanation algorithms have been proposed to address this issue, yet limited research effort has been reported concerning their user evaluation. In this paper, we report on an online between-group user study designed to evaluate the performance of "saliency maps" - a popular explanation algorithm for image classification applications of CNNs. Our results indicate that saliency maps produced by the LRP algorithm helped participants to learn about some specific image features the system is sensitive to. However, the maps seem to provide very limited help for participants to anticipate the network's output for new images. Drawing on our findings, we highlight implications for design and further research on explainable AI. In particular, we argue the HCI and AI communities should look beyond instance-level explanations.