# Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms

arXiv: https://arxiv.org/abs/1910.07387 · HTML: https://mlsanity.com/paper/1910.07387/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 30 of 32)

- [Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models](https://mlsanity.com/paper/2502.12892/) (2025)
- [Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysis](https://mlsanity.com/paper/2202.07728/) (2022)
- [Missingness Bias in Model Debugging](https://mlsanity.com/paper/2204.08945/) (2022)
- [COVID-Net CT-2: Enhanced Deep Neural Networks for Detection of COVID-19 from Chest CT Images Through Bigger, More Diverse Learning](https://mlsanity.com/paper/2101.07433/) (2021)
- [Xplique: A Deep Learning Explainability Toolbox](https://mlsanity.com/paper/2206.04394/) (2022)
- [COVID-Net CXR-2: An Enhanced Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-ray Images](https://mlsanity.com/paper/2105.06640/) (2021)
- [Evaluating Feature Attribution Methods in the Image Domain](https://mlsanity.com/paper/2202.12270/) (2022)
- [Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognition](https://mlsanity.com/paper/2305.18011/) (2023)
- [TB-Net: A Tailored, Self-Attention Deep Convolutional Neural Network Design for Detection of Tuberculosis Cases from Chest X-ray Images](https://mlsanity.com/paper/2104.03165/) (2021)
- [An adversarial attack approach for eXplainable AI evaluation on deepfake detection models](https://mlsanity.com/paper/2312.06627/) (2023)
- [COVID-Net CXR-S: Deep Convolutional Neural Network for Severity Assessment of COVID-19 Cases from Chest X-ray Images](https://mlsanity.com/paper/2105.00256/) (2021)
- [A Unified Study of Machine Learning Explanation Evaluation Metrics](https://mlsanity.com/paper/2203.14265/) (2022)
- [On the explainable properties of 1-Lipschitz Neural Networks: An Optimal Transport Perspective](https://mlsanity.com/paper/2206.06854/) (2022)
- [COVID-Net USPro: An Open-Source Explainable Few-Shot Deep Prototypical Network to Monitor and Detect COVID-19 Infection from Point-of-Care Ultrasound Images](https://mlsanity.com/paper/2301.01679/) (2023)
- [COVID-Net US: A Tailored, Highly Efficient, Self-Attention Deep Convolutional Neural Network Design for Detection of COVID-19 Patient Cases from Point-of-care Ultrasound Imaging](https://mlsanity.com/paper/2108.03131/) (2021)
- [TinyDefectNet: Highly Compact Deep Neural Network Architecture for High-Throughput Manufacturing Visual Quality Inspection](https://mlsanity.com/paper/2111.14319/) (2021)
- [Fibrosis-Net: A Tailored Deep Convolutional Neural Network Design for Prediction of Pulmonary Fibrosis Progression from Chest CT Images](https://mlsanity.com/paper/2103.04008/) (2021)
- [COVID-Net Clinical ICU: Enhanced Prediction of ICU Admission for COVID-19 Patients via Explainability and Trust Quantification](https://mlsanity.com/paper/2109.06711/) (2021)
- [Topological Representations of Local Explanations](https://mlsanity.com/paper/2201.02155/) (2022)
- [SolderNet: Towards Trustworthy Visual Inspection of Solder Joints in Electronics Manufacturing Using Explainable Artificial Intelligence](https://mlsanity.com/paper/2211.10274/) (2022)
- [Vision Transformer for Efficient Chest X-ray and Gastrointestinal Image Classification](https://mlsanity.com/paper/2304.11529/) (2023)
- [TrustyAI Explainability Toolkit](https://mlsanity.com/paper/2104.12717/) (2021)
- [Insights into Data through Model Behaviour: An Explainability-driven Strategy for Data Auditing for Responsible Computer Vision Applications](https://mlsanity.com/paper/2106.09177/) (2021)
- [COVID-Net Biochem: An Explainability-driven Framework to Building Machine Learning Models for Predicting Survival and Kidney Injury of COVID-19 Patients from Clinical and Biochemistry Data](https://mlsanity.com/paper/2204.11210/) (2022)
- [On the Bias-Variance Characteristics of LIME and SHAP in High Sparsity Movie Recommendation Explanation Tasks](https://mlsanity.com/paper/2206.04784/) (2022)
- [Interactive Diabetes Risk Prediction Using Explainable Machine Learning: A Dash-Based Approach with SHAP, LIME, and Comorbidity Insights](https://mlsanity.com/paper/2505.05683/) (2025)
- [COVID-Net MLSys: Designing COVID-Net for the Clinical Workflow](https://mlsanity.com/paper/2109.06421/) (2021)
- [POTHER: Patch-Voted Deep Learning-Based Chest X-ray Bias Analysis for COVID-19 Detection](https://mlsanity.com/paper/2201.09360/) (2022)
- [Towards Trustworthy Healthcare AI: Attention-Based Feature Learning for COVID-19 Screening With Chest Radiography](https://mlsanity.com/paper/2207.09312/) (2022)
- [Explaining Explainability: Towards Deeper Actionable Insights into Deep Learning through Second-order Explainability](https://mlsanity.com/paper/2306.08780/) (2023)

## Abstract (arXiv)

There has been a significant surge of interest recently around the concept of explainable artificial intelligence (XAI), where the goal is to produce an interpretation for a decision made by a machine learning algorithm. Of particular interest is the interpretation of how deep neural networks make decisions, given the complexity and `black box' nature of such networks. Given the infancy of the field, there has been very limited exploration into the assessment of the performance of explainability methods, with most evaluations centered around subjective visual interpretation of the produced interpretations. In this study, we explore a more machine-centric strategy for quantifying the performance of explainability methods on deep neural networks via the notion of decision-making impact analysis. We introduce two quantitative performance metrics: i) Impact Score, which assesses the percentage of critical factors with either strong confidence reduction impact or decision changing impact, and ii) Impact Coverage, which assesses the percentage coverage of adversarially impacted factors in the input. A comprehensive analysis using this approach was conducted on several state-of-the-art explainability methods (LIME, SHAP, Expected Gradients, GSInquire) on a ResNet-50 deep convolutional neural network using a subset of ImageNet for the task of image classification. Experimental results show that the critical regions identified by LIME within the tested images had the lowest impact on the decision-making process of the network (~38%), with progressive increase in decision-making impact for SHAP (~44%), Expected Gradients (~51%), and GSInquire (~76%). While by no means perfect, the hope is that the proposed machine-centric strategy helps push the conversation forward towards better metrics for evaluating explainability methods and improve trust in deep neural networks.