Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing
Not on the map. Why?
Abstract (arXiv)
While enjoying the great achievements brought by deep learning (DL), people are also worried about the decision made by DL models, since the high degree of non-linearity of DL models makes the decision extremely difficult to understand. Consequently, attacks such as adversarial attacks are easy to carry out, but difficult to detect and explain, which has led to a boom in the research on local explanation methods for explaining model decisions. In this paper, we evaluate the faithfulness of explanation methods and find that traditional tests on faithfulness encounter the random dominance problem, ie, the random selection performs the best, especially for complex data. To further solve this problem, we propose three trend-based faithfulness tests and empirically demonstrate that the new trend tests can better assess faithfulness than traditional tests on image, natural language and security tasks. We implement the assessment system and evaluate ten popular explanation methods. Benefiting from the trend tests, we successfully assess the explanation methods on complex data for the first time, bringing unprecedented discoveries and inspiring future research. Downstream tasks also greatly benefit from the tests. For example, model debugging equipped with faithful explanation methods performs much better for detecting and correcting accuracy and security problems.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 48 of its 77 references that are in the map. First posted on arXiv 9 Sep 2023.
Builds on
48 of 77 references.
References · 48
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2016Densely Connected Convolutional Networkscited by 6,720
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
- 2017Axiomatic Attribution for Deep Networkscited by 3,384
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
36 more
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
- 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
- 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
- 2018Sanity Checks for Saliency Mapscited by 1,156
- 2017SmoothGrad: removing noise by adding noisecited by 1,109
- 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
- 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
- 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
- 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
- 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
- 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
- 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
- 2020Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Surveycited by 254
- 2017Real Time Image Saliency for Black Box Classifierscited by 246
- 2019Explanations can be manipulated and geometry is to blamecited by 216
- 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 211
- 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
- 2020Debugging Tests for Model Explanationscited by 147
- 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
- 2021Deep Learning Through the Lens of Example Difficultycited by 141
- 2019Benchmarking Attribution Methods with Relative Feature Importancecited by 89
- 2022Understanding the unstable convergence of gradient descentcited by 86
- 2018Interpretable Deep Learning under Firecited by 74
- 2018How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanationcited by 72
- 2021Model Complexity of Deep Learning: A Surveycited by 65
- 2021Vulnerability Detection with Fine-grained Interpretationscited by 56
- 2020Evaluations and Methods for Explanation through Robustness Analysiscited by 51
- 2015Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Featurescited by 48
- 2018Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Valuescited by 40
- 2019Evaluating Explanation Methods for Deep Learning in Securitycited by 38
- 2018Human-in-the-Loop Interpretability Priorcited by 36
- 2021Adversarial Robustness of Deep Code Comment Generationcited by 22
- 2021Hidden Backdoor Attack against Semantic Segmentation Modelscited by 20
- 2022Quarantine: Sparsity Can Uncover the Trojan Attack Trigger for Freecited by 15
- 2021Interpreting Attributions and Interactions of Adversarial Attackscited by 13
- 2022"Is your explanation stable?": A Robustness Evaluation Framework for Feature Attributioncited by 7
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 1.
2 papers.