# Developing a Fidelity Evaluation Approach for Interpretable Machine Learning

arXiv: https://arxiv.org/abs/2106.08492 · HTML: https://mlsanity.com/paper/2106.08492/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies](https://mlsanity.com/paper/2007.15911/) (2020)
- [RuleMatrix: Visualizing and Understanding Classifiers with Rules](https://mlsanity.com/paper/1807.06228/) (2018)
- [ALIME: Autoencoder Based Approach for Local Interpretability](https://mlsanity.com/paper/1909.02437/) (2019)
- [Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead](https://mlsanity.com/paper/1811.10154/) (2018)
- [Statistical stability indices for LIME: obtaining reliable explanations for Machine Learning models](https://mlsanity.com/paper/2001.11757/) (2020)
- [Evaluating Explanation Without Ground Truth in Interpretable Machine Learning](https://mlsanity.com/paper/1907.06831/) (2019)
- [A Human-Grounded Evaluation of SHAP for Alert Processing](https://mlsanity.com/paper/1907.03324/) (2019)
- [On Attribution of Recurrent Neural Network Predictions via Additive Decomposition](https://mlsanity.com/paper/1903.11245/) (2019)
- [Evaluating the Correctness of Explainable AI Algorithms for Classification](https://mlsanity.com/paper/2105.09740/) (2021)
- [An Interpretable Probabilistic Approach for Demystifying Black-box Predictive Models](https://mlsanity.com/paper/2007.10668/) (2020)

## Built on by (most cited first, 8 of 8)

- [Framework for Evaluating Faithfulness of Local Explanations](https://mlsanity.com/paper/2202.00734/) (2022)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Understanding Disparities in Post Hoc Machine Learning Explanation](https://mlsanity.com/paper/2401.14539/) (2024)
- [A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2307.14246/) (2023)
- [Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI](https://mlsanity.com/paper/2308.06274/) (2023)
- [RESHAPE: Explaining Accounting Anomalies in Financial Statement Audits by enhancing SHapley Additive exPlanations](https://mlsanity.com/paper/2209.09157/) (2022)
- [Weakly Supervised Pixel-Level Annotation with Visual Interpretability](https://mlsanity.com/paper/2502.17824/) (2025)
- [Human-Centered Explainable AI for TinyML Edge Devices: A Pareto-Based Selection Framework with LLM-Guided Design](https://mlsanity.com/paper/2608.07091/) (2026)

## Abstract (arXiv)

Although modern machine learning and deep learning methods allow for complex and in-depth data analytics, the predictive models generated by these methods are often highly complex, and lack transparency. Explainable AI (XAI) methods are used to improve the interpretability of these complex models, and in doing so improve transparency. However, the inherent fitness of these explainable methods can be hard to evaluate. In particular, methods to evaluate the fidelity of the explanation to the underlying black box require further development, especially for tabular data. In this paper, we (a) propose a three phase approach to developing an evaluation method; (b) adapt an existing evaluation method primarily for image and text data to evaluate models trained on tabular data; and (c) evaluate two popular explainable methods using this evaluation method. Our evaluations suggest that the internal mechanism of the underlying predictive model, the internal mechanism of the explainable method used and model and data complexity all affect explanation fidelity. Given that explanation fidelity is so sensitive to context and tools and data used, we could not clearly identify any specific explainable method as being superior to another.