# Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values

arXiv: https://arxiv.org/abs/1810.03307 · HTML: https://mlsanity.com/paper/1810.03307/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2018.

## Built on by (most cited first, 30 of 40)

- [Metrics for saliency map evaluation of deep learning explanation methods](https://mlsanity.com/paper/2201.13291/) (2022)
- [SwiFT: Swin 4D fMRI Transformer](https://mlsanity.com/paper/2307.05916/) (2023)
- [Auxiliary Losses for Learning Generalizable Concept-based Models](https://mlsanity.com/paper/2311.11108/) (2023)
- [Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution](https://mlsanity.com/paper/2108.01070/) (2021)
- [Opti-CAM: Optimizing saliency maps for interpretability](https://mlsanity.com/paper/2301.07002/) (2023)
- [Shap-CAM: Visual Explanations for Convolutional Neural Networks based on Shapley Value](https://mlsanity.com/paper/2208.03608/) (2022)
- [ODAM: Gradient-based instance-specific visual explanations for object detection](https://mlsanity.com/paper/2304.06354/) (2023)
- [B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable](https://mlsanity.com/paper/2411.00715/) (2024)
- [Making Corgis Important for Honeycomb Classification: Adversarial Attacks on Concept-based Explainability Tools](https://mlsanity.com/paper/2110.07120/) (2021)
- [Cause and Effect: Hierarchical Concept-based Explanation of Neural Networks](https://mlsanity.com/paper/2105.07033/) (2021)
- [Occlusion Sensitivity Analysis with Augmentation Subspace Perturbation in Deep Feature Space](https://mlsanity.com/paper/2311.15022/) (2023)
- [Toward Understanding the Disagreement Problem in Neural Network Feature Attribution](https://mlsanity.com/paper/2404.11330/) (2024)
- [A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference](https://mlsanity.com/paper/2204.05428/) (2022)
- [On the Robustness of Explanations of Deep Neural Network Models: A Survey](https://mlsanity.com/paper/2211.04780/) (2022)
- [A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2307.14246/) (2023)
- [Exploring Explainability in Video Action Recognition](https://mlsanity.com/paper/2404.09067/) (2024)
- [Expected Grad-CAM: Towards gradient faithfulness](https://mlsanity.com/paper/2406.01274/) (2024)
- [Finding Representative Interpretations on Convolutional Neural Networks](https://mlsanity.com/paper/2108.06384/) (2021)
- [Accelerating Multi-Objective Neural Architecture Search by Random-Weight Evaluation](https://mlsanity.com/paper/2110.05242/) (2021)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [On Pitfalls of textitRemOve-And-Retrain: Data Processing Inequality Perspective](https://mlsanity.com/paper/2304.13836/) (2023)
- [BR-NPA: A Non-Parametric High-Resolution Attention Model to improve the Interpretability of Attention](https://mlsanity.com/paper/2106.02566/) (2021)
- [Data augmentation and explainability for bias discovery and mitigation in deep learning](https://mlsanity.com/paper/2308.09464/) (2023)
- [Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing](https://mlsanity.com/paper/2309.05679/) (2023)
- [CA-Stream: Attention-based pooling for interpretable image recognition](https://mlsanity.com/paper/2404.14996/) (2024)
- [Global explainability in aligned image modalities](https://mlsanity.com/paper/2112.09591/) (2021)
- [Geometrically Guided Integrated Gradients](https://mlsanity.com/paper/2206.05903/) (2022)
- [Contrastive Pretraining for Visual Concept Explanations of Socioeconomic Outcomes](https://mlsanity.com/paper/2404.09768/) (2024)
- [Flow AM: Generating Point Cloud Global Explanations by Latent Alignment](https://mlsanity.com/paper/2404.18760/) (2024)
- [DeepLeak: Privacy Enhancing Hardening of Model Explanations Against Membership Leakage](https://mlsanity.com/paper/2601.03429/) (2026)

## Abstract (arXiv)

Explaining the output of a complicated machine learning model like a deep neural network (DNN) is a central challenge in machine learning. Several proposed local explanation methods address this issue by identifying what dimensions of a single input are most responsible for a DNN's output. The goal of this work is to assess the sensitivity of local explanations to DNN parameter values. Somewhat surprisingly, we find that DNNs with randomly-initialized weights produce explanations that are both visually and quantitatively similar to those produced by DNNs with learned weights. Our conjecture is that this phenomenon occurs because these explanations are dominated by the lower level features of a DNN, and that a DNN's architecture provides a strong prior which significantly affects the representations learned at these lower layers. NOTE: This work is now subsumed by our recent manuscript, Sanity Checks for Saliency Maps (to appear NIPS 2018), where we expand on findings and address concerns raised in Sundararajan et. al. (2018).