# Framework for Evaluating Faithfulness of Local Explanations

arXiv: https://arxiv.org/abs/2202.00734 · HTML: https://mlsanity.com/paper/2202.00734/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 59 references that are in the map.
- First posted on arXiv 1 Feb 2022.

## Builds on (most cited first, 30 of 38)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)
- [Evaluating and Aggregating Feature-based Model Explanations](https://mlsanity.com/paper/2005.00631/) (2020)
- [Sanity Checks for Saliency Metrics](https://mlsanity.com/paper/1912.01451/) (2019)
- [Counterfactual Explanations Can Be Manipulated](https://mlsanity.com/paper/2106.02666/) (2021)
- [Optimal Sparse Decision Trees](https://mlsanity.com/paper/1904.12847/) (2019)
- [Evaluating Explanations: How much do explanations from the teacher aid students?](https://mlsanity.com/paper/2012.00893/) (2020)
- [The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations](https://mlsanity.com/paper/2106.00786/) (2021)
- [Aligning Faithful Interpretations with their Social Attribution](https://mlsanity.com/paper/2006.01067/) (2020)
- [Model Interpretability through the Lens of Computational Complexity](https://mlsanity.com/paper/2010.12265/) (2020)
- [Fairwashing Explanations with Off-Manifold Detergent](https://mlsanity.com/paper/2007.09969/) (2020)
- [Explainable k-Means and k-Medians Clustering](https://mlsanity.com/paper/2002.12538/) (2020)
- [A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations](https://mlsanity.com/paper/1805.07039/) (2018)
- [How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations](https://mlsanity.com/paper/2101.08758/) (2021)
- [Towards falsifiable interpretability research](https://mlsanity.com/paper/2010.12016/) (2020)
- [Learning Optimized Risk Scores](https://mlsanity.com/paper/1610.00168/) (2016)
- [Robust and Stable Black Box Explanations](https://mlsanity.com/paper/2011.06169/) (2020)
- [Towards the Unification and Robustness of Perturbation and Gradient Based Explanations](https://mlsanity.com/paper/2102.10618/) (2021)
- [Gradient-based Analysis of NLP Models is Manipulable](https://mlsanity.com/paper/2010.05419/) (2020)

## Built on by (most cited first, 30 of 55)

- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [OpenXAI: Towards a Transparent Evaluation of Model Explanations](https://mlsanity.com/paper/2206.11104/) (2022)
- [Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals](https://mlsanity.com/paper/2310.00603/) (2023)
- [Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency](https://mlsanity.com/paper/2306.02109/) (2023)
- [The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus](https://mlsanity.com/paper/2302.07265/) (2023)
- [MultiViz: Towards Visualizing and Understanding Multimodal Models](https://mlsanity.com/paper/2207.00056/) (2022)
- [SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretability](https://mlsanity.com/paper/2208.09418/) (2022)
- [Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics](https://mlsanity.com/paper/2409.16756/) (2024)
- [Feature Perturbation Augmentation for Reliable Evaluation of Importance Estimators in Neural Networks](https://mlsanity.com/paper/2303.01538/) (2023)
- [A Framework for Interpretability in Machine Learning for Medical Imaging](https://mlsanity.com/paper/2310.01685/) (2023)
- [Sanity Checks Revisited: An Exploration to Repair the Model Parameter Randomisation Test](https://mlsanity.com/paper/2401.06465/) (2024)
- [Intrinsic User-Centric Interpretability through Global Mixture of Experts](https://mlsanity.com/paper/2402.02933/) (2024)
- [A Fresh Look at Sanity Checks for Saliency Maps](https://mlsanity.com/paper/2405.02383/) (2024)
- [Auditing Local Explanations is Hard](https://mlsanity.com/paper/2407.13281/) (2024)
- [Benchmarking XAI Explanations with Human-Aligned Evaluations](https://mlsanity.com/paper/2411.02470/) (2024)
- [Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models](https://mlsanity.com/paper/2311.01732/) (2023)
- [Toward Understanding the Disagreement Problem in Neural Network Feature Attribution](https://mlsanity.com/paper/2404.11330/) (2024)
- [XAudit : A Theoretical Look at Auditing with Explanations](https://mlsanity.com/paper/2206.04740/) (2022)
- [GNN Explanations that do not Explain and How to find Them](https://mlsanity.com/paper/2601.20815/) (2026)
- [A Survey of Explainable AI in Deep Visual Modeling: Methods and Metrics](https://mlsanity.com/paper/2301.13445/) (2023)
- [Expected Grad-CAM: Towards gradient faithfulness](https://mlsanity.com/paper/2406.01274/) (2024)
- [From Flexibility to Manipulation: The Slippery Slope of XAI Evaluation](https://mlsanity.com/paper/2412.05592/) (2024)
- [Are explainable AI (XAI) evaluation strategies aligned? Comparing subjective, objective, and mathematical evaluation measures using saliency maps](https://mlsanity.com/paper/2504.17023/) (2025)
- [IBO: Inpainting-Based Occlusion to Enhance Explainable Artificial Intelligence Evaluation in Histopathology](https://mlsanity.com/paper/2408.16395/) (2024)
- [Faithful and Robust Local Interpretability for Textual Predictions](https://mlsanity.com/paper/2311.01605/) (2023)
- [A Sim2Real Approach for Identifying Task-Relevant Properties in Interpretable Machine Learning](https://mlsanity.com/paper/2406.00116/) (2024)
- [DISCRET: Synthesizing Faithful Explanations For Treatment Effect Estimation](https://mlsanity.com/paper/2406.00611/) (2024)
- [GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifiers](https://mlsanity.com/paper/2411.15605/) (2024)
- [ExpProof : Operationalizing Explanations for Confidential Models with ZKPs](https://mlsanity.com/paper/2502.03773/) (2025)

## Abstract (arXiv)

We study the faithfulness of an explanation system to the underlying prediction model. We show that this can be captured by two properties, consistency and sufficiency, and introduce quantitative measures of the extent to which these hold. Interestingly, these measures depend on the test-time data distribution. For a variety of existing explanation systems, such as anchors, we analytically study these quantities. We also provide estimators and sample complexity bounds for empirically determining the faithfulness of black-box explanation systems. Finally, we experimentally validate the new properties and estimators.