Data to 14 Oct 2026

The (Un)reliability of saliency methods

arXiv 1711.00867 ↗First posted 2 Nov 2017stat.ML cs.LGcited by 333

Not on the map. Why?

Abstract (arXiv)

Saliency methods aim to explain the predictions of deep neural networks. These methods lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction. We use a simple and common pre-processing step ---adding a constant shift to the input data--- to show that a transformation with no effect on the model can cause numerous methods to incorrectly attribute. In order to guarantee reliability, we posit that methods should fulfill input invariance, the requirement that a saliency method mirror the sensitivity of the model with respect to transformations of the input. We show, through several examples, that saliency methods that do not satisfy input invariance result in misleading attribution.

Key facts. Cited by 333 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 57; 2022: 73; 2023: 62; 2024: 56; 2025: 48; 2026: 36.

333 papers.

Sort:

By area:

2021Causal Abstractions of Neural Networkscited by 422
2023Explainability for Large Language Models: A Surveycited by 333
2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 178
2022Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyondcited by 171
2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
2021Acquisition of Chess Knowledge in AlphaZerocited by 123
2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 109
2023Concept-based Explainable Artificial Intelligence: A Surveycited by 98
2023Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglementcited by 90
2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
2021Explaining Time Series Predictions with Dynamic Maskscited by 73

38 more

2021Improving Deep Learning Interpretability by Saliency Guided Trainingcited by 68
2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
2022Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Offcited by 61
2022Additive MIL: Intrinsically Interpretable Multiple Instance Learning for Pathologycited by 58
2022Concept Activation Regions: A Generalized Framework For Concept-Based Explanationscited by 57
2023Interpretable Neural-Symbolic Concept Reasoningcited by 57
2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 53
2021Do Input Gradients Highlight Discriminative Features?cited by 52
2021Entropy-based Logic Explanations of Neural Networkscited by 51
2021Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networkscited by 48
2022Logic-Based Explainability in Machine Learningcited by 48
2023Adversarial attacks and defenses in explainable artificial intelligence: A surveycited by 48
2022Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capabilitycited by 46
2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 46
2023Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistencycited by 42
2021A Survey on the Robustness of Feature Importance and Counterfactual Explanationscited by 39
2021Combining Feature and Instance Attribution to Detect Artifactscited by 38
2022Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysiscited by 38
2022Missingness Bias in Model Debuggingcited by 38
2022Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overviewcited by 37
2023Neuron-Level Knowledge Attribution in Large Language Modelscited by 37
2021Order in the Court: Explainable AI Methods Prone to Disagreementcited by 36
2021Zorro: Valid, Sparse, and Stable Explanations in Graph Neural Networkscited by 35
2021GCExplainer: Human-in-the-Loop Concept-based Explanations for Graph Neural Networkscited by 35
2024LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivitycited by 33
2024Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robustcited by 33
2023Quantifying Uncertainty in Natural Language Explanations of Large Language Modelscited by 32
2021Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retrainingcited by 31
2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31
2021On the Sensitivity and Stability of Model Interpretations in NLPcited by 30
2021EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use casecited by 30
2021Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behaviorcited by 29
2021NoiseGrad: Enhancing Explanations by Introducing Stochasticity to Model Weightscited by 28
2021CAMERAS: Enhanced Resolution And Sanity preserving Class Activation Mapping for image saliencycited by 28
2022Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geosciencecited by 28
2022Interpretable by Design: Learning Predictors by Composing Interpretable Queriescited by 28
2022SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretabilitycited by 28
2021Taxonomy of Machine Learning Safety: A Survey and Primercited by 25

50 most cited of 333 All 333 (JSON)