# One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images

arXiv: https://arxiv.org/abs/2107.05047 · HTML: https://mlsanity.com/paper/2107.05047/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 60 references that are in the map.
- First posted on arXiv 11 Jul 2021.

## Builds on (most cited first, 18 of 18)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use](https://mlsanity.com/paper/1905.05134/) (2019)
- [Explainable deep learning models in medical image analysis](https://mlsanity.com/paper/2005.13799/) (2020)
- [InfoMask: Masked Variational Latent Representation to Localize Chest Disease](https://mlsanity.com/paper/1903.11741/) (2019)
- [Automatic brain tumor grading from MRI data using convolutional neural networks and quality assessment](https://mlsanity.com/paper/1809.09468/) (2018)
- [Artificial Intelligence in Glioma Imaging: Challenges and Advances](https://mlsanity.com/paper/1911.12886/) (2019)
- [Efficient Interpretation of Deep Learning Models Using Graph Structure and Cooperative Game Theory: Application to ASD Biomarker Discovery](https://mlsanity.com/paper/1812.06181/) (2018)

## Built on by (most cited first, 11 of 11)

- [Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods](https://mlsanity.com/paper/2111.02398/) (2021)
- [Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysis](https://mlsanity.com/paper/2202.10553/) (2022)
- [Attention-based Saliency Maps Improve Interpretability of Pneumothorax Classification](https://mlsanity.com/paper/2303.01871/) (2023)
- [AI-Based Detection, Classification and Prediction/Prognosis in Medical Imaging: Towards Radiophenomics](https://mlsanity.com/paper/2110.10332/) (2021)
- [End-To-End Prediction of Knee Osteoarthritis Progression With Multi-Modal Transformers](https://mlsanity.com/paper/2307.00873/) (2023)
- [Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks](https://mlsanity.com/paper/2511.05883/) (2025)
- [Feature-level Interaction Explanations in Multimodal Transformers](https://mlsanity.com/paper/2603.13326/) (2026)
- [Towards the Use of Saliency Maps for Explaining Low-Quality Electrocardiograms to End Users](https://mlsanity.com/paper/2207.02726/) (2022)
- [Influence based explainability of brain tumors segmentation in multimodal Magnetic Resonance Imaging](https://mlsanity.com/paper/2405.12222/) (2024)
- [Saliency Guided Longitudinal Medical Visual Question Answering](https://mlsanity.com/paper/2509.25374/) (2025)
- [Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models](https://mlsanity.com/paper/2606.06534/) (2026)

## Abstract (arXiv)

Being able to explain the prediction to clinical end-users is a necessity to leverage the power of AI models for clinical decision support. For medical images, saliency maps are the most common form of explanation. The maps highlight important features for AI model's prediction. Although many saliency map methods have been proposed, it is unknown how well they perform on explaining decisions on multi-modal medical images, where each modality/channel carries distinct clinical meanings of the same underlying biomedical phenomenon. Understanding such modality-dependent features is essential for clinical users' interpretation of AI decisions. To tackle this clinically important but technically ignored problem, we propose the MSFI (Modality-Specific Feature Importance) metric to examine whether saliency maps can highlight modality-specific important features. MSFI encodes the clinical requirements on modality prioritization and modality-specific feature localization. Our evaluations on 16 commonly used saliency map methods, including a clinician user study, show that although most saliency map methods captured modality importance information in general, most of them failed to highlight modality-specific important features consistently and precisely. The evaluation results guide the choices of saliency map methods and provide insights to propose new ones targeting clinical applications.