# Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods

arXiv: https://arxiv.org/abs/1910.02065 · HTML: https://mlsanity.com/paper/1910.02065/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2019.

## Built on by (most cited first, 30 of 38)

- [Faithfulness Tests for Natural Language Explanations](https://mlsanity.com/paper/2305.18029/) (2023)
- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts](https://mlsanity.com/paper/2201.10295/) (2022)
- [Logic-Based Explainability in Machine Learning](https://mlsanity.com/paper/2211.00541/) (2022)
- [Prompting Contrastive Explanations for Commonsense Reasoning Tasks](https://mlsanity.com/paper/2106.06823/) (2021)
- [On Tackling Explanation Redundancy in Decision Trees](https://mlsanity.com/paper/2205.09971/) (2022)
- [Towards Formal XAI: Formally Approximate Minimal Explanations of Neural Networks](https://mlsanity.com/paper/2210.13915/) (2022)
- [SAT-Based Rigorous Explanations for Decision Lists](https://mlsanity.com/paper/2105.06782/) (2021)
- [On Efficiently Explaining Graph-Based Classifiers](https://mlsanity.com/paper/2106.01350/) (2021)
- [Order in the Court: Explainable AI Methods Prone to Disagreement](https://mlsanity.com/paper/2105.03287/) (2021)
- [Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations](https://mlsanity.com/paper/2106.13876/) (2021)
- [Formally Explaining Neural Networks within Reactive Systems](https://mlsanity.com/paper/2308.00143/) (2023)
- [Eliminating The Impossible, Whatever Remains Must Be True](https://mlsanity.com/paper/2206.09551/) (2022)
- [On Computing Probabilistic Abductive Explanations](https://mlsanity.com/paper/2212.05990/) (2022)
- [Pixel-Grounded Prototypical Part Networks](https://mlsanity.com/paper/2309.14531/) (2023)
- [Correctness is not Faithfulness in RAG Attributions](https://mlsanity.com/paper/2412.18004/) (2024)
- [Efficient Explanations With Relevant Sets](https://mlsanity.com/paper/2106.00546/) (2021)
- [Clairvoyance: A Pipeline Toolkit for Medical Time Series](https://mlsanity.com/paper/2310.18688/) (2023)
- [Towards a consistent interpretation of AIOps models](https://mlsanity.com/paper/2202.02298/) (2022)
- [On Computing Relevant Features for Explaining NBCs](https://mlsanity.com/paper/2207.04748/) (2022)
- [Self-Explaining Neural Networks for Business Process Monitoring](https://mlsanity.com/paper/2503.18067/) (2025)
- [Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods](https://mlsanity.com/paper/2310.05619/) (2023)
- [A Uniform Language to Explain Decision Trees](https://mlsanity.com/paper/2310.11636/) (2023)
- [Towards Faithful Explanations for Text Classification with Robustness Improvement and Explanation Guided Training](https://mlsanity.com/paper/2312.17591/) (2023)
- [A Framework for Evaluating Post Hoc Feature-Additive Explainers](https://mlsanity.com/paper/2106.08376/) (2021)
- [The Role of Syntactic Span Preferences in Post-Hoc Explanation Disagreement](https://mlsanity.com/paper/2403.19424/) (2024)
- [Interpretability in Symbolic Regression: a benchmark of Explanatory Methods using the Feynman data set](https://mlsanity.com/paper/2404.05908/) (2024)
- [Benchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language Tasks](https://mlsanity.com/paper/2304.08174/) (2023)
- [How Well Do Feature-Additive Explainers Explain Feature-Additive Predictors?](https://mlsanity.com/paper/2310.18496/) (2023)
- [On the Complexity-Faithfulness Trade-off of Gradient-Based Explanations](https://mlsanity.com/paper/2508.10490/) (2025)

## Abstract (arXiv)

For AI systems to garner widespread public acceptance, we must develop methods capable of explaining the decisions of black-box models such as neural networks. In this work, we identify two issues of current explanatory methods. First, we show that two prevalent perspectives on explanations --- feature-additivity and feature-selection --- lead to fundamentally different instance-wise explanations. In the literature, explainers from different perspectives are currently being directly compared, despite their distinct explanation goals. The second issue is that current post-hoc explainers are either validated under simplistic scenarios (on simple models such as linear regression, or on models trained on syntactic datasets), or, when applied to real-world neural networks, explainers are commonly validated under the assumption that the learned models behave reasonably. However, neural networks often rely on unreasonable correlations, even when producing correct decisions. We introduce a verification framework for explanatory methods under the feature-selection perspective. Our framework is based on a non-trivial neural network architecture trained on a real-world task, and for which we are able to provide guarantees on its inner workings. We validate the efficacy of our evaluation by showing the failure modes of current explainers. We aim for this framework to provide a publicly available, off-the-shelf evaluation when the feature-selection perspective on explanations is needed.