How Well Do Feature-Additive Explainers Explain Feature-Additive Predictors?
Not on the map. Why?
Abstract (arXiv)
Surging interest in deep learning from high-stakes domains has precipitated concern over the inscrutable nature of black box neural networks. Explainable AI (XAI) research has led to an abundance of explanation algorithms for these black boxes. Such post hoc explainers produce human-comprehensible explanations, however, their fidelity with respect to the model is not well understood - explanation evaluation remains one of the most challenging issues in XAI. In this paper, we ask a targeted but important question: can popular feature-additive explainers (e.g., LIME, SHAP, SHAPR, MAPLE, and PDP) explain feature-additive predictors? Herein, we evaluate such explainers on ground truth that is analytically derived from the additive structure of a model. We demonstrate the efficacy of our approach in understanding these explainers applied to symbolic expressions, neural networks, and generalized additive models on thousands of synthetic and several real-world tasks. Our results suggest that all explainers eventually fail to correctly attribute the importance of features, especially when a decision-making process involves feature interactions.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 69 of its 130 references that are in the map. First posted on arXiv 27 Oct 2023.
Builds on
69 of 130 references.
References · 69
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
- 2013Intriguing properties of neural networkscited by 5,155
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
- 2019Natural Adversarial Examplescited by 1,408
- 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
- 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
- 2019GNNExplainer: Generating Explanations for Graph Neural Networkscited by 737
- 2016The Mythos of Model Interpretabilitycited by 721
- 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
- 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 443
38 more
- 2020Parameterized Explainer for Graph Neural Networkcited by 426
- 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 404
- 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 319
- 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 302
- 2017Right for the Right Reasons: Training Differentiable Models by Constraining their Explanationscited by 294
- 2018Learning to Explain: An Information-Theoretic Perspective on Model Interpretationcited by 248
- 2019Explanations can be manipulated and geometry is to blamecited by 216
- 2020PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networkscited by 211
- 2019Explaining individual predictions when features are dependent: More accurate approximations to Shapley valuescited by 208
- 2022The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspectivecited by 159
- 2020Debugging Tests for Model Explanationscited by 147
- 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
- 2019Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainabilitycited by 114
- 2019Learning to Deceive with Attention-Based Explanationscited by 111
- 2020Benchmarking Deep Learning Interpretability in Time Series Predictionscited by 102
- 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
- 2020Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Databasecited by 96
- 2020On the Tractability of SHAP Explanationscited by 92
- 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 91
- 2018Interpreting CNNs via Decision Treescited by 89
- 2020Explaining the Explainer: A First Theoretical Analysis of LIMEcited by 89
- 2021CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithmscited by 86
- 2018Model Agnostic Supervised Local Explanationscited by 83
- 2020How does this interaction affect me? Interpretable attribution for feature interactionscited by 73
- 2022From Shapley Values to Generalized Additive Models and backcited by 60
- 2018What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Playcited by 53
- 2020Towards falsifiable interpretability researchcited by 53
- 2022Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contextscited by 53
- 2019Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networkscited by 49
- 2019Measurable Counterfactual Local Explanations for Any Classifiercited by 48
- 2017Visual Feature Attribution using Wasserstein GANscited by 47
- 2020Robust and Stable Black Box Explanationscited by 47
- 2022e-CARE: a New Dataset for Exploring Explainable Causal Reasoningcited by 47
- 2019Can I Trust the Explainer? Verifying Post-hoc Explanatory Methodscited by 38
- 2023The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantuscited by 37
- 2019Grid Saliency for Context Explanations of Semantic Segmentationcited by 32
- 2023Don't trust your eyes: on the (un)reliability of feature visualizationscited by 32
- 2017Visual Explanation by Interpretation: Improving Visual Feedback Capabilities of Deep Neural Networkscited by 28
50 most cited of 69 All 69 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.