mlsanityData to 14 Oct 2026

Sanity Checks for Saliency Maps

arXiv 1810.03292 ↗First posted 8 Oct 2018cs.CV cs.LG stat.MLcited by 1,156

Not on the map. Why?

Cited by 26 in Mechanistic interpretability, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, finding outliers in the data, explaining the relationship between inputs and outputs that the model learned, and debugging the model. We interpret our findings through an analogy with edge detection in images, a technique that requires neither training data nor model. Theory in the case of a linear model and a single-layer convolutional neural network supports our experimental findings.

Key facts. Cited by 1,156 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 189; 2022: 209; 2023: 217; 2024: 175; 2025: 175; 2026: 191.

1,156 papers.

Sort:

By area:

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  2. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  3. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 436
  4. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 386
  5. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 367
  6. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  7. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 302
  8. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 178
  9. 2022Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyondcited by 171
  10. 2022The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspectivecited by 159
  11. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  12. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 156
38 more
  1. 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
  2. 2021Explainable artificial intelligence (XAI) in deep learning-based medical image analysiscited by 147
  3. 2021Benchmarking and Survey of Explanation Methods for Black Box Modelscited by 136
  4. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  5. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
  6. 2021Acquisition of Chess Knowledge in AlphaZerocited by 123
  7. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 118
  8. 2022A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainabilitycited by 118
  9. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  10. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  11. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  12. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 109
  13. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  14. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  15. 2022A Consistent and Efficient Evaluation Strategy for Attribution Methodscited by 102
  16. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  17. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  18. 2023Concept-based Explainable Artificial Intelligence: A Surveycited by 98
  19. 2021Interpretable Image Classification with Differentiable Prototypes Assignmentcited by 97
  20. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  21. 2022Trustworthy Graph Neural Networks: Aspects, Methods and Trendscited by 89
  22. 2024What makes unlearning hard and what to do about itcited by 89
  23. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 88
  24. 2022B-cos Networks: Alignment is All We Need for Interpretabilitycited by 88
  25. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 86
  26. 2021Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methodscited by 83
  27. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  28. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  29. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  30. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 80
  31. 2022Concept Bottleneck Model with Additional Unsupervised Conceptscited by 79
  32. 2022Impossibility Theorems for Feature Attributioncited by 78
  33. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  34. 2022Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanationscited by 77
  35. 2021XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanationscited by 73
  36. 2021Explaining Time Series Predictions with Dynamic Maskscited by 73
  37. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 73
  38. 2021IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammographycited by 72

50 most cited of 1,156 All 1,156 (JSON)