# Impossibility Theorems for Feature Attribution

arXiv: https://arxiv.org/abs/2212.11870 · HTML: https://mlsanity.com/paper/2212.11870/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 22 Dec 2022.

## Builds on (most cited first, 30 of 34)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Explaining individual predictions when features are dependent: More accurate approximations to Shapley values](https://mlsanity.com/paper/1903.10464/) (2019)
- [Problems with Shapley-value-based explanations as feature importance measures](https://mlsanity.com/paper/2002.11097/) (2020)
- [Full-Gradient Representation for Neural Network Visualization](https://mlsanity.com/paper/1905.00780/) (2019)
- [Feature relevance quantification in explainable AI: A causal problem](https://mlsanity.com/paper/1910.13413/) (2019)
- [XRAI: Better Attributions Through Regions](https://mlsanity.com/paper/1906.02825/) (2019)
- [Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations](https://mlsanity.com/paper/2206.01254/) (2022)
- [A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations](https://mlsanity.com/paper/1805.07039/) (2018)
- [Visualizing Deep Networks by Optimizing with Integrated Gradients](https://mlsanity.com/paper/1905.00954/) (2019)
- [Local Explanations via Necessity and Sufficiency: Unifying Theory and Practice](https://mlsanity.com/paper/2103.14651/) (2021)
- [One Explanation is Not Enough: Structured Attention Graphs for Image Classification](https://mlsanity.com/paper/2011.06733/) (2020)
- [Approximating Lipschitz continuous functions with GroupSort neural networks](https://mlsanity.com/paper/2006.05254/) (2020)
- [Improved Bounds on Neural Complexity for Representing Piecewise Linear Functions](https://mlsanity.com/paper/2210.07236/) (2022)
- [Using Attribution to Decode Dataset Bias in Neural Network Models for Chemistry](https://mlsanity.com/paper/1811.11310/) (2018)
- [Attribution-based Explanations that Provide Recourse Cannot be Robust](https://mlsanity.com/paper/2205.15834/) (2022)
- [iGOS++: Integrated Gradient Optimized Saliency by Bilateral Perturbations](https://mlsanity.com/paper/2012.15783/) (2020)
- [FairCanary: Rapid Continuous Explainable Fairness](https://mlsanity.com/paper/2106.07057/) (2021)
- [Towards Rigorous Interpretations: a Formalisation of Feature Attribution](https://mlsanity.com/paper/2104.12437/) (2021)

## Built on by (most cited first, 30 of 78)

- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [LLM Circuit Analyses Are Consistent Across Training and Scale](https://mlsanity.com/paper/2407.10827/) (2024)
- [Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review](https://mlsanity.com/paper/2402.10086/) (2024)
- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [Explaining Predictive Uncertainty with Information Theoretic Shapley Values](https://mlsanity.com/paper/2306.05724/) (2023)
- [Don't trust your eyes: on the (un)reliability of feature visualizations](https://mlsanity.com/paper/2306.04719/) (2023)
- [Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars](https://mlsanity.com/paper/2312.01429/) (2023)
- [Rethinking Data Shapley for Data Selection Tasks: Misleads and Merits](https://mlsanity.com/paper/2405.03875/) (2024)
- [We Can't Understand AI Using our Existing Vocabulary](https://mlsanity.com/paper/2502.07586/) (2025)
- [The Interpretable and Effective Graph Neural Additive Networks](https://mlsanity.com/paper/2406.01317/) (2024)
- [Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations](https://mlsanity.com/paper/2504.12691/) (2025)
- [Stability Guarantees for Feature Attributions with Multiplicative Smoothing](https://mlsanity.com/paper/2307.05902/) (2023)
- [LLMScan: Causal Scan for LLM Misbehavior Detection](https://mlsanity.com/paper/2410.16638/) (2024)
- [Towards Interpretable Deep Generative Models via Causal Representation Learning](https://mlsanity.com/paper/2504.11609/) (2025)
- [From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolution](https://mlsanity.com/paper/2404.06647/) (2024)
- [The Dead Salmons of AI Interpretability](https://mlsanity.com/paper/2512.18792/) (2025)
- [Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Survey](https://mlsanity.com/paper/2405.01636/) (2024)
- [Interpretability Needs a New Paradigm](https://mlsanity.com/paper/2405.05386/) (2024)
- [Sample based Explanations via Generalized Representers](https://mlsanity.com/paper/2310.18526/) (2023)
- [Probabilistic Stability Guarantees for Feature Attributions](https://mlsanity.com/paper/2504.13787/) (2025)
- [Using Reinforcement Learning to Train Large Language Models to Explain Human Decisions](https://mlsanity.com/paper/2505.11614/) (2025)
- [Learning with Explanation Constraints](https://mlsanity.com/paper/2303.14496/) (2023)
- [Rethinking Explainable Machine Learning as Applied Statistics](https://mlsanity.com/paper/2402.02870/) (2024)
- [Multi-view Disentanglement for Reinforcement Learning with Multiple Cameras](https://mlsanity.com/paper/2404.14064/) (2024)
- [Feature Responsiveness Scores: Model-Agnostic Explanations for Recourse](https://mlsanity.com/paper/2410.22598/) (2024)
- [Causality is Key for Interpretability Claims to Generalise](https://mlsanity.com/paper/2602.16698/) (2026)
- [Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups](https://mlsanity.com/paper/2310.16316/) (2023)
- [Provably Better Explanations with Optimized Aggregation of Feature Attributions](https://mlsanity.com/paper/2406.05090/) (2024)
- [FLEXtime: Filterbank learning to explain time series](https://mlsanity.com/paper/2411.05841/) (2024)
- [Mask of truth: model sensitivity to unexpected regions of medical images](https://mlsanity.com/paper/2412.04030/) (2024)

## Abstract (arXiv)

Despite a sea of interpretability methods that can produce plausible explanations, the field has also empirically seen many failure cases of such methods. In light of these results, it remains unclear for practitioners how to use these methods and choose between them in a principled way. In this paper, we show that for moderately rich model classes (easily satisfied by neural networks), any feature attribution method that is complete and linear -- for example, Integrated Gradients and SHAP -- can provably fail to improve on random guessing for inferring model behaviour. Our results apply to common end-tasks such as characterizing local model behaviour, identifying spurious features, and algorithmic recourse. One takeaway from our work is the importance of concretely defining end-tasks: once such an end-task is defined, a simple and direct approach of repeated model evaluations can outperform many other complex feature attribution methods.