mlsanityData to 7 Oct 2026

On the Robustness of Explanations of Deep Neural Network Models: A Survey

arXiv 2211.04780 ↗First posted 9 Nov 2022cs.LG cs.CR cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Explainability has been widely stated as a cornerstone of the responsible and trustworthy use of machine learning models. With the ubiquitous use of Deep Neural Network (DNN) models expanding to risk-sensitive and safety-critical domains, many methods have been proposed to explain the decisions of these models. Recent years have also seen concerted efforts that have shown how such explanations can be distorted (attacked) by minor input perturbations. While there have been many surveys that review explainability methods themselves, there has been no effort hitherto to assimilate the different methods and metrics proposed to study the robustness of explanations of DNN models. In this work, we present a comprehensive survey of methods that study, understand, attack, and defend explanations of DNN models. We also present a detailed review of different metrics used to evaluate explanation methods, as well as describe attributional attack and defense methods. We conclude with lessons and take-aways for the community towards ensuring robust explanations of DNN model predictions.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 91 of its 133 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

91 of 133 references.

References · 91

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  8. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  9. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  10. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  11. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  12. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
38 more
  1. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  2. 2018Sanity Checks for Saliency Mapscited by 1,149
  3. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  4. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  5. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
  6. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  7. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  8. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  9. 2020Overfitting in adversarially robust deep learningcited by 504
  10. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
  11. 2017Interpretation of Neural Networks is Fragilecited by 443
  12. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 440
  13. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
  14. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 403
  15. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  16. 2017The (Un)reliability of saliency methodscited by 331
  17. 2016Top-down Neural Attention by Excitation Backpropcited by 318
  18. 2018On the Robustness of Interpretability Methodscited by 310
  19. 2019On the (In)fidelity and Sensitivity for Explanationscited by 309
  20. 2020Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Surveycited by 252
  21. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  22. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
  23. 2019Explanations can be manipulated and geometry is to blamecited by 215
  24. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 197
  25. 2020Evaluating and Aggregating Feature-based Model Explanationscited by 185
  26. 2018Robustness via curvature regularization, and vice versacited by 174
  27. 2018Adversarial Attack and Defense on Graph Data: A Surveycited by 166
  28. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  29. 2019Fooling Neural Network Interpretations via Adversarial Model Manipulationcited by 112
  30. 2019"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanationscited by 111
  31. 2019On the Connection Between Adversarial Robustness and Saliency Map Interpretabilitycited by 80
  32. 2019Neural Network Attributions: A Causal Perspectivecited by 77
  33. 2018Interpretable Deep Learning under Firecited by 74
  34. 2018How Important Is a Neuron?cited by 73
  35. 2020Opportunities and Challenges in Deep Learning Adversarial Robustness: A Surveycited by 71
  36. 2021Explainable Artificial Intelligence (XAI) on TimeSeries Data: A Surveycited by 71
  37. 2018Did the Model Understand the Question?cited by 60
  38. 2021A Survey On Universal Adversarial Attackcited by 59

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2024Revisiting the robustness of post-hoc interpretability methodscited by 5
  2. 2025Hierarchical Testing with Rabbit Optimization for Industrial Cyber-Physical Systemscited by 4
  3. 2025Accuracy is Not Enough: Poisoning Interpretability in Federated Learning via Color Skewcited by 0
  4. 2025Hybrid Attribution Priors for Explainable and Robust Model Trainingcited by 0
  5. 2026Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?cited by 0