mlsanityData to 7 Oct 2026

Impossibility Theorems for Feature Attribution

arXiv 2212.11870 ↗First posted 22 Dec 2022cs.LG cs.AIcited by 78

Not on the map. Why?

Abstract (arXiv)

Despite a sea of interpretability methods that can produce plausible explanations, the field has also empirically seen many failure cases of such methods. In light of these results, it remains unclear for practitioners how to use these methods and choose between them in a principled way. In this paper, we show that for moderately rich model classes (easily satisfied by neural networks), any feature attribution method that is complete and linear -- for example, Integrated Gradients and SHAP -- can provably fail to improve on random guessing for inferring model behaviour. Our results apply to common end-tasks such as characterizing local model behaviour, identifying spurious features, and algorithmic recourse. One takeaway from our work is the importance of concretely defining end-tasks: once such an end-task is defined, a simple and direct approach of repeated model evaluations can outperform many other complex feature attribution methods.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 22 Dec 2022.

Builds on

34 of 57 references.

References · 34

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  6. 2018Sanity Checks for Saliency Mapscited by 1,149
  7. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  8. 2020Concept Bottleneck Modelscited by 1,022
  9. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  10. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  11. 2019The many Shapley values for model explanationcited by 341
  12. 2019On the (In)fidelity and Sensitivity for Explanationscited by 309
22 more
  1. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  2. 2019Explaining individual predictions when features are dependent: More accurate approximations to Shapley valuescited by 205
  3. 2020Problems with Shapley-value-based explanations as feature importance measurescited by 198
  4. 2019Full-Gradient Representation for Neural Network Visualizationcited by 189
  5. 2019Feature relevance quantification in explainable AI: A causal problemcited by 164
  6. 2019XRAI: Better Attributions Through Regionscited by 116
  7. 2022Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanationscited by 77
  8. 2018A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizationscited by 59
  9. 2019Visualizing Deep Networks by Optimizing with Integrated Gradientscited by 59
  10. 2021Local Explanations via Necessity and Sufficiency: Unifying Theory and Practicecited by 45
  11. 2020One Explanation is Not Enough: Structured Attention Graphs for Image Classificationcited by 32
  12. 2020Approximating Lipschitz continuous functions with GroupSort neural networkscited by 27
  13. 2022Improved Bounds on Neural Complexity for Representing Piecewise Linear Functionscited by 26
  14. 2018Using Attribution to Decode Dataset Bias in Neural Network Models for Chemistrycited by 25
  15. 2022Attribution-based Explanations that Provide Recourse Cannot be Robustcited by 20
  16. 2020iGOS++: Integrated Gradient Optimized Saliency by Bilateral Perturbationscited by 19
  17. 2021FairCanary: Rapid Continuous Explainable Fairnesscited by 12
  18. 2021Towards Rigorous Interpretations: a Formalisation of Feature Attributioncited by 11
  19. 2022The Solvability of Interpretability Evaluation Metricscited by 11
  20. 2020Biased Models Have Biased Explanationscited by 9
  21. 2018A Feature Selection Method Based on Shapley Value to False Alarm Reduction in ICUs, A Genetic-Algorithm Approachcited by 7
  22. 2021Symmetry-Preserving Paths in Integrated Gradientscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 26; 2025: 23; 2026: 16.

78 papers.

Sort:

By area:

  1. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  2. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  3. 2024Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Reviewcited by 52
  4. 2023Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZerocited by 41
  5. 2023Explaining Predictive Uncertainty with Information Theoretic Shapley Valuescited by 36
  6. 2023Don't trust your eyes: on the (un)reliability of feature visualizationscited by 32
  7. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  8. 2024Rethinking Data Shapley for Data Selection Tasks: Misleads and Meritscited by 29
  9. 2025We Can't Understand AI Using our Existing Vocabularycited by 25
  10. 2024The Interpretable and Effective Graph Neural Additive Networkscited by 19
  11. 2023Stability Guarantees for Feature Attributions with Multiplicative Smoothingcited by 14
  12. 2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 14
38 more
  1. 2024LLMScan: Causal Scan for LLM Misbehavior Detectioncited by 13
  2. 2024From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolutioncited by 12
  3. 2025Towards Interpretable Deep Generative Models via Causal Representation Learningcited by 12
  4. 2024Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Surveycited by 11
  5. 2023Sample based Explanations via Generalized Representerscited by 10
  6. 2024Interpretability Needs a New Paradigmcited by 10
  7. 2025The Dead Salmons of AI Interpretabilitycited by 10
  8. 2025Probabilistic Stability Guarantees for Feature Attributionscited by 9
  9. 2023Learning with Explanation Constraintscited by 8
  10. 2024Rethinking Explainable Machine Learning as Applied Statisticscited by 8
  11. 2024Multi-view Disentanglement for Reinforcement Learning with Multiple Camerascited by 8
  12. 2024Feature Responsiveness Scores: Model-Agnostic Explanations for Recoursecited by 8
  13. 2025Using Reinforcement Learning to Train Large Language Models to Explain Human Decisionscited by 8
  14. 2026Causality is Key for Interpretability Claims to Generalisecited by 8
  15. 2023Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groupscited by 6
  16. 2024Provably Better Explanations with Optimized Aggregation of Feature Attributionscited by 6
  17. 2024FLEXtime: Filterbank learning to explain time seriescited by 6
  18. 2024Mask of truth: model sensitivity to unexpected regions of medical imagescited by 6
  19. 2024Explainable AI needs formalizationcited by 4
  20. 2026TreeGrad-Ranker: Feature Ranking via O(L)-Time Gradients for Decision Treescited by 4
  21. 2024Explaining models relating objects and privacycited by 3
  22. 2024Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributionscited by 3
  23. 2025Can Global XAI Methods Reveal Injected Behaviours in LLMs? SHAP vs Rule Extraction vs RuleSHAPcited by 3
  24. 2025Local MDI+: Local Feature Importances for Tree-Based Modelscited by 3
  25. 2023SPADE: Sparsity-Guided Debugging for Deep Neural Networkscited by 2
  26. 2024People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AIcited by 2
  27. 2024Less Discriminatory Alternative and Interpretable XGBoost Framework for Binary Classificationcited by 2
  28. 2025Informative Post-Hoc Explanations Only Exist for Simple Functionscited by 2
  29. 2026Manifold-Aligned Guided Integrated Gradients for Reliable Feature Attributioncited by 2
  30. 2023Is Task-Agnostic Explainable AI a Myth?cited by 1
  31. 2023Measuring Information in Text Explanationscited by 1
  32. 2024Prospector Heads: Generalized Feature Attribution for Large Models & Datacited by 1
  33. 2024MiMICRI: Towards Domain-centered Counterfactual Explanations of Cardiovascular Image Classification Modelscited by 1
  34. 2024Data Science Principles for Interpretable and Explainable AIcited by 1
  35. 2024In Defence of Post-hoc Explainabilitycited by 1
  36. 2025Towards Unified Attribution in Explainable AI, Data-Centric AI, and Mechanistic Interpretabilitycited by 1
  37. 2025How to safely discard features based on aggregate SHAP valuescited by 1
  38. 2025Fast Fourier Correlation is a Highly Efficient and Accurate Feature Attribution Algorithm from the Perspective of Control Theory and Game Theorycited by 1

50 most cited of 78 All 78 (JSON)