mlsanityData to 7 Oct 2026

Do Feature Attribution Methods Correctly Attribute Features?

arXiv 2104.14403 ↗First posted 27 Apr 2021cs.LG cs.CVcited by 98

Not on the map. Why?

Abstract (arXiv)

Feature attribution methods are popular in interpretable machine learning. These methods compute the attribution of each input feature to represent its importance, but there is no consensus on the definition of "attribution", leading to many competing methods with little systematic evaluation, complicated in particular by the lack of ground truth attribution. To address this, we propose a dataset modification procedure to induce such ground truth. Using this procedure, we evaluate three common methods: saliency maps, rationales, and attentions. We identify several deficiencies and add new perspectives to the growing body of evidence questioning the correctness and reliability of these methods applied on datasets in the wild. We further discuss possible avenues for remedy and recommend new attribution methods to be tested against ground truth before deployment. The code is available at https://github.com/YilunZhou/feature-attribution-evaluation

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 27 Apr 2021.

Builds on

35 of 48 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  8. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  11. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  12. 2018Sanity Checks for Saliency Mapscited by 1,149
23 more
  1. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  2. 2019Attention is not Explanationcited by 916
  3. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  4. 2019Attention is not not Explanationcited by 629
  5. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  6. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  7. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  8. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  9. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
  10. 2016Rationalizing Neural Predictionscited by 281
  11. 2020Debugging Tests for Model Explanationscited by 146
  12. 2019Interpretable Neural Predictions with Differentiable Binary Variablescited by 126
  13. 2020Learning to Faithfully Rationalize by Constructioncited by 113
  14. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  15. 2020Benchmarking Deep Learning Interpretability in Time Series Predictionscited by 102
  16. 2019Rethinking Cooperative Rationalization: Introspective Extraction and Complement Controlcited by 96
  17. 2019Benchmarking Attribution Methods with Relative Feature Importancecited by 89
  18. 2017End-to-end Training for Whole Image Breast Cancer Diagnosis using An All Convolutional Designcited by 87
  19. 2020Aligning Faithful Interpretations with their Social Attributioncited by 79
  20. 2019Evaluating Recurrent Neural Network Explanationscited by 29
  21. 2020Deep Learning Applied to Chest X-Rays: Exploiting and Preventing Shortcutscited by 29
  22. 2019Convolutional neural network models for cancer type prediction based on gene expressioncited by 22
  23. 2020Understanding Global Feature Contributions With Additive Importance Measurescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 18; 2023: 32; 2024: 18; 2025: 13; 2026: 9.

98 papers.

Sort:

By area:

  1. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  2. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  3. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  4. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  5. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
  6. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  7. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  8. 2022A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributescited by 56
  9. 2022Learning to Estimate Shapley Values with Vision Transformerscited by 44
  10. 2021Combining Feature and Instance Attribution to Detect Artifactscited by 38
  11. 2022Missingness Bias in Model Debuggingcited by 36
  12. 2022Can Rationalization Improve Robustness?cited by 36
38 more
  1. 2022Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysiscited by 34
  2. 2021How Well do Feature Visualizations Support Causal Understanding of CNN Activations?cited by 28
  3. 2022Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geosciencecited by 28
  4. 2023Finding the right XAI method -- A Guide for the Evaluation and Ranking of Explainable AI Methods in Climate Sciencecited by 25
  5. 2021Revisiting Sanity Checks for Saliency Mapscited by 20
  6. 2021The Irrationality of Neural Rationale Modelscited by 16
  7. 2023Studying How to Efficiently and Effectively Guide Models with Explanationscited by 16
  8. 2021Sanity Simulations for Saliency Methodscited by 15
  9. 2022Diagnosing AI Explanation Methods with Folk Concepts of Behaviorcited by 15
  10. 2022ExSum: From Local Explanations to Model Understandingcited by 15
  11. 2023Are Data-driven Explanations Robust against Out-of-distribution Data?cited by 14
  12. 2023Stability Guarantees for Feature Attributions with Multiplicative Smoothingcited by 14
  13. 2024B-cosification: Transforming Deep Neural Networks to be Inherently Interpretablecited by 12
  14. 2022The Solvability of Interpretability Evaluation Metricscited by 11
  15. 2024Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Surveycited by 11
  16. 2023Quantifying and Explaining Machine Learning Uncertainty in Predictive Process Monitoring: An Operations Research Perspectivecited by 10
  17. 2023Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognitioncited by 10
  18. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  19. 2023Is Last Layer Re-Training Truly Sufficient for Robustness to Spurious Correlations?cited by 9
  20. 2025Probabilistic Stability Guarantees for Feature Attributionscited by 9
  21. 2024Comprehensible Artificial Intelligence on Knowledge Graphs: A surveycited by 8
  22. 2024Feature Responsiveness Scores: Model-Agnostic Explanations for Recoursecited by 8
  23. 2023XAI-TRIS: Non-linear image benchmarks to quantify false positive post-hoc attribution of feature importancecited by 7
  24. 2023Error Discovery by Clustering Influence Embeddingscited by 7
  25. 2022Visualizing Automatic Speech Recognition -- Means for a Better Understanding?cited by 6
  26. 2023Generative Perturbation Analysis for Probabilistic Black-Box Anomaly Attributioncited by 6
  27. 2024What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blockscited by 6
  28. 2024Provably Better Explanations with Optimized Aggregation of Feature Attributionscited by 6
  29. 2022Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behaviorcited by 5
  30. 2024Toward Understanding the Disagreement Problem in Neural Network Feature Attributioncited by 5
  31. 2024On the Evaluation Consistency of Attribution-based Explanationscited by 5
  32. 2023Neighboring Words Affect Human Interpretation of Saliency Explanationscited by 4
  33. 2023Neuron Patching: Semantic-based Neuron-level Language Model Repair for Code Generationcited by 4
  34. 2024Explainable AI needs formalizationcited by 4
  35. 2021A Framework for Evaluating Post Hoc Feature-Additive Explainerscited by 3
  36. 2022Robustness of Explanation Methods for NLP Modelscited by 3
  37. 2023AttributionLab: Faithfulness of Feature Attribution Under Controllable Environmentscited by 3
  38. 2023Evaluating the Utility of Model Explanations for Model Developmentcited by 3

50 most cited of 98 All 98 (JSON)