Informative Post-Hoc Explanations Only Exist for Simple Functions
Not on the map. Why?
Abstract (arXiv)
Many researchers have suggested that local post-hoc explanation algorithms can be used to gain insights into the behavior of complex machine learning models. However, theoretical guarantees about such algorithms only exist for simple decision functions, and it is unclear whether and under which assumptions similar results might exist for complex models. In this paper, we introduce a general, learning-theory-based framework for what it means for an explanation to provide information about a decision function. We call an explanation informative if it serves to reduce the complexity of the space of plausible decision functions. With this approach, we show that many popular explanation algorithms are not informative when applied to complex decision functions, providing a rigorous mathematical rejection of the idea that it should be possible to explain any model. We then derive conditions under which different explanation algorithms become informative. These are often stronger than what one might expect. For example, gradient explanations and counterfactual explanations are non-informative with respect to the space of differentiable functions, and SHAP and anchor explanations are not informative with respect to the space of decision trees. Based on these results, we discuss how explanation algorithms can be modified to become informative. While the proposed analysis of explanation algorithms is mathematical, we argue that it holds strong implications for the practical applicability of these algorithms, particularly for auditing, regulation, and high-risk applications of AI.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 42 of its 61 references that are in the map. First posted on arXiv 15 Aug 2025.
Builds on
42 of 61 references.
References · 42
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2017Axiomatic Attribution for Deep Networkscited by 3,384
- 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
- 2018Sanity Checks for Saliency Mapscited by 1,156
- 2017SmoothGrad: removing noise by adding noisecited by 1,109
- 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,089
- 2018Robustness May Be at Odds with Accuracycited by 880
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
- 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 443
- 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 302
30 more
- 2020Problems with Shapley-value-based explanations as feature importance measurescited by 200
- 2020Evaluating and Aggregating Feature-based Model Explanationscited by 188
- 2022The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspectivecited by 159
- 2020Debugging Tests for Model Explanationscited by 147
- 2015Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction modelcited by 140
- 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
- 2021Counterfactual Explanations Can Be Manipulatedcited by 113
- 2020Explaining the Explainer: A First Theoretical Analysis of LIMEcited by 89
- 2022Impossibility Theorems for Feature Attributioncited by 78
- 2022Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanationscited by 77
- 2020General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Modelscited by 75
- 2020Fairwashing Explanations with Off-Manifold Detergentcited by 72
- 2022From Shapley Values to Generalized Additive Models and backcited by 60
- 2022Framework for Evaluating Faithfulness of Local Explanationscited by 55
- 2022Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contextscited by 53
- 2019Are Perceptually-Aligned Gradients a General Property of Robust Classifiers?cited by 43
- 2022What Makes a Good Explanation?: A Harmonized View of Properties of Explanationscited by 22
- 2018Automated Test Generation to Detect Individual Discrimination in AI Modelscited by 21
- 2021Robust Models Are More Interpretable Because Attributions Look Normalcited by 21
- 2022Attribution-based Explanations that Provide Recourse Cannot be Robustcited by 20
- 2023Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustnesscited by 20
- 2022Efficiently Training Low-Curvature Neural Networkscited by 19
- 2025Beyond Topological Self-Explainable GNNs: A Formal Explainability Perspectivecited by 11
- 2025InstaSHAP: Interpretable Additive Models Explain Shapley Values Instantlycited by 9
- 2024Rethinking Explainable Machine Learning as Applied Statisticscited by 8
- 2024Auditing Local Explanations is Hardcited by 8
- 2024A Theory of Interpretable Approximationscited by 7
- 2022On the explainable properties of 1-Lipschitz Neural Networks: An Optimal Transport Perspectivecited by 6
- 2022Boundary-Aware Uncertainty for Feature Attribution Explainerscited by 6
- 2024X Hacking: The Threat of Misguided AutoMLcited by 5
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.