mlsanityData to 14 Oct 2026

Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability

arXiv 2108.01335 ↗First posted 3 Aug 2021cs.CV cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Conventional saliency maps highlight input features to which neural network predictions are highly sensitive. We take a different approach to saliency, in which we identify and analyze the network parameters, rather than inputs, which are responsible for erroneous decisions. We find that samples which cause similar parameters to malfunction are semantically similar. We also show that pruning the most salient parameters for a wrongly classified sample often improves model behavior. Furthermore, fine-tuning a small number of the most salient parameters on a single sample results in error correction on other samples that are misclassified for similar reasons. Based on our parameter saliency method, we also introduce an input-space saliency technique that reveals how image features cause specific network components to malfunction. Further, we rigorously validate the meaningfulness of our saliency maps on both the dataset and case-study levels.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 43 of its 65 references that are in the map. First posted on arXiv 3 Aug 2021.

Builds on

43 of 65 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  8. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  11. 2019Language Models as Knowledge Bases?cited by 1,936
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
31 more
  1. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  3. 2018Sanity Checks for Saliency Mapscited by 1,156
  4. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  5. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  7. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  8. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  9. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  10. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
  11. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
  12. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  13. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  14. 2015Understanding Neural Networks Through Deep Visualizationcited by 334
  15. 2016Top-down Neural Attention by Excitation Backpropcited by 321
  16. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 297
  17. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  18. 2019Full-Gradient Representation for Neural Network Visualizationcited by 191
  19. 2020How Context Affects Language Models' Factual Predictionscited by 181
  20. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  21. 2017Interpreting Deep Visual Representations via Network Dissectioncited by 103
  22. 2018On the importance of single directions for generalizationcited by 99
  23. 2019Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruningcited by 89
  24. 2017Explaining Recurrent Neural Network Predictions in Sentiment Analysiscited by 88
  25. 2016A deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstructioncited by 87
  26. 2020Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Studycited by 86
  27. 2017meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfittingcited by 34
  28. 2021Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behaviorcited by 29
  29. 2020Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruptioncited by 26
  30. 2020Utilizing Explainable AI for Quantization and Pruning of Deep Neural Networkscited by 17
  31. 2017Interpreting Convolutional Neural Networks Through Compressioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 0; 2023: 2; 2024: 2; 2025: 0; 2026: 1.

6 papers.

  1. 2023A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learningcited by 24
  2. 2024Talking Nonsense: Probing Large Language Models' Understanding of Adversarial Gibberish Inputscited by 11
  3. 2021Identification of Attack-Specific Signatures in Adversarial Examplescited by 4
  4. 2024KS-Lottery: Finding Certified Lottery Tickets for Multilingual Language Modelscited by 3
  5. 2023Understanding Parameter Saliency via Extreme Value Theorycited by 0
  6. 2026Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistancecited by 0