# Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?

arXiv: https://arxiv.org/abs/2203.06487 · HTML: https://mlsanity.com/paper/2203.06487/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 57 references that are in the map.
- First posted on arXiv 12 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Visualizing Deep Neural Network Decisions: Prediction Difference Analysis](https://mlsanity.com/paper/1702.04595/) (2017)
- [Explainable deep learning models in medical image analysis](https://mlsanity.com/paper/2005.13799/) (2020)
- [Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches](https://mlsanity.com/paper/1912.05100/) (2019)
- [InfoMask: Masked Variational Latent Representation to Localize Chest Disease](https://mlsanity.com/paper/1903.11741/) (2019)
- [Artificial Intelligence in Glioma Imaging: Challenges and Advances](https://mlsanity.com/paper/1911.12886/) (2019)
- [EUCA: the End-User-Centered Explainable AI Framework](https://mlsanity.com/paper/2102.02437/) (2021)

## Built on by (most cited first, 20 of 20)

- [Explainability in AI Policies: A Critical Review of Communications, Reports, Regulations, and Standards in the EU, US, and UK](https://mlsanity.com/paper/2304.11218/) (2023)
- [MProtoNet: A Case-Based Interpretable Model for Brain Tumor Classification with 3D Multi-parametric Magnetic Resonance Imaging](https://mlsanity.com/paper/2304.06258/) (2023)
- [Interpreting and Correcting Medical Image Classification with PIP-Net](https://mlsanity.com/paper/2307.10404/) (2023)
- [Why is plausibility surprisingly problematic as an XAI criterion?](https://mlsanity.com/paper/2303.17707/) (2023)
- [Prototype-Based Interpretability for Legal Citation Prediction](https://mlsanity.com/paper/2305.16490/) (2023)
- [MRxaI: Black-Box Explainability for Image Classifiers in a Medical Setting](https://mlsanity.com/paper/2311.14471/) (2023)
- [Invisible Users: Uncovering End-Users' Requirements for Explainable AI via Explanation Forms and Goals](https://mlsanity.com/paper/2302.06609/) (2023)
- [Transcending XAI Algorithm Boundaries through End-User-Inspired Design](https://mlsanity.com/paper/2208.08739/) (2022)
- [Automatic Infectious Disease Classification Analysis with Concept Discovery](https://mlsanity.com/paper/2209.02415/) (2022)
- [What are You Looking at? Modality Contribution in Multimodal Medical Deep Learning](https://mlsanity.com/paper/2503.01904/) (2025)
- [Consistent and Invariant Generalization Learning for Short-video Misinformation Detection](https://mlsanity.com/paper/2507.04061/) (2025)
- [On Faithfulness and Coherence of Language Explanations for Recommendation Systems](https://mlsanity.com/paper/2209.05409/) (2022)
- [Clinical Domain Knowledge-Derived Template Improves Post Hoc AI Explanations in Pneumothorax Classification](https://mlsanity.com/paper/2403.18871/) (2024)
- [Order-Preserving Dimension Reduction for Multimodal Semantic Embedding](https://mlsanity.com/paper/2408.10264/) (2024)
- [Interactivity x Explainability: Toward Understanding How Interactivity Can Improve Computer Vision Explanations](https://mlsanity.com/paper/2504.10745/) (2025)
- [Explainable Artificial Intelligence in Biomedical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2507.07148/) (2025)
- [Influence based explainability of brain tumors segmentation in multimodal Magnetic Resonance Imaging](https://mlsanity.com/paper/2405.12222/) (2024)
- [PRECISe : Prototype-Reservation for Explainable Classification under Imbalanced and Scarce-Data Settings](https://mlsanity.com/paper/2408.05754/) (2024)
- [Bridging AI and Clinical Reasoning: Abductive Explanations for Alignment on Critical Symptoms](https://mlsanity.com/paper/2602.13985/) (2026)
- [Architecture-aware Robustness Evaluation of Explainable Deep Learning for Breast Cancer Diagnosis](https://mlsanity.com/paper/2609.31692/) (2026)

## Abstract (arXiv)

Being able to explain the prediction to clinical end-users is a necessity to leverage the power of artificial intelligence (AI) models for clinical decision support. For medical images, a feature attribution map, or heatmap, is the most common form of explanation that highlights important features for AI models' prediction. However, it is unknown how well heatmaps perform on explaining decisions on multi-modal medical images, where each image modality or channel visualizes distinct clinical information of the same underlying biomedical phenomenon. Understanding such modality-dependent features is essential for clinical users' interpretation of AI decisions. To tackle this clinically important but technically ignored problem, we propose the modality-specific feature importance (MSFI) metric. It encodes clinical image and explanation interpretation patterns of modality prioritization and modality-specific feature localization. We conduct a clinical requirement-grounded, systematic evaluation using computational methods and a clinician user study. Results show that the examined 16 heatmap algorithms failed to fulfill clinical requirements to correctly indicate AI model decision process or decision quality. The evaluation and MSFI metric can guide the design and selection of XAI algorithms to meet clinical requirements on multi-modal explanation.