# Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging

arXiv: https://arxiv.org/abs/2008.02766 · HTML: https://mlsanity.com/paper/2008.02766/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2020.

## Built on by (most cited first, 30 of 32)

- [Explainable artificial intelligence (XAI) in deep learning-based medical image analysis](https://mlsanity.com/paper/2107.10912/) (2021)
- [Fair Conformal Predictors for Applications in Medical Imaging](https://mlsanity.com/paper/2109.04392/) (2021)
- [IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography](https://mlsanity.com/paper/2103.12308/) (2021)
- [MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentation](https://mlsanity.com/paper/2409.19483/) (2024)
- [MedCLIP-SAM: Bridging Text and Image Towards Universal Medical Image Segmentation](https://mlsanity.com/paper/2403.20253/) (2024)
- [Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysis](https://mlsanity.com/paper/2202.10553/) (2022)
- [Evaluating Feature Attribution Methods in the Image Domain](https://mlsanity.com/paper/2202.12270/) (2022)
- [Attri-VAE: attribute-based interpretable representations of medical images with variational autoencoders](https://mlsanity.com/paper/2203.10417/) (2022)
- [The Manifold Hypothesis for Gradient-Based Explanations](https://mlsanity.com/paper/2206.07387/) (2022)
- [Sanity Simulations for Saliency Methods](https://mlsanity.com/paper/2105.06506/) (2021)
- [Medical Imaging and Machine Learning](https://mlsanity.com/paper/2103.01938/) (2021)
- [Saliency Cards: A Framework to Characterize and Compare Saliency Methods](https://mlsanity.com/paper/2206.02958/) (2022)
- [Interpretable Mammographic Image Classification using Case-Based Reasoning and Deep Learning](https://mlsanity.com/paper/2107.05605/) (2021)
- [Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognition](https://mlsanity.com/paper/2305.18011/) (2023)
- [Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions](https://mlsanity.com/paper/2107.09546/) (2021)
- [On the Robustness of Explanations of Deep Neural Network Models: A Survey](https://mlsanity.com/paper/2211.04780/) (2022)
- [Optimising Knee Injury Detection with Spatial Attention and Validating Localisation Ability](https://mlsanity.com/paper/2108.08136/) (2021)
- [Rethinking Robustness of Model Attributions](https://mlsanity.com/paper/2312.10534/) (2023)
- [Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays](https://mlsanity.com/paper/2102.09475/) (2021)
- [CheXseg: Combining Expert Annotations with DNN-generated Saliency Maps for X-ray Segmentation](https://mlsanity.com/paper/2102.10484/) (2021)
- [Interpretability Aware Model Training to Improve Robustness against Out-of-Distribution Magnetic Resonance Images in Alzheimer's Disease Classification](https://mlsanity.com/paper/2111.08701/) (2021)
- [Multi-modal volumetric concept activation to explain detection and classification of metastatic prostate cancer on PSMA-PET/CT](https://mlsanity.com/paper/2208.02555/) (2022)
- [Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models](https://mlsanity.com/paper/2308.08774/) (2023)
- [Representing visual classification as a linear combination of words](https://mlsanity.com/paper/2311.10933/) (2023)
- [Prospector Heads: Generalized Feature Attribution for Large Models & Data](https://mlsanity.com/paper/2402.11729/) (2024)
- [Exploiting XAI maps to improve MS lesion segmentation and detection in MRI](https://mlsanity.com/paper/2409.03772/) (2024)
- [A Quantitative Approach for Evaluating Disease Focus and Interpretability of Deep Learning Models for Alzheimer's Disease Classification](https://mlsanity.com/paper/2409.04888/) (2024)
- [When saliency goes off on a tangent: Interpreting Deep Neural Networks with nonlinear saliency maps](https://mlsanity.com/paper/2110.06639/) (2021)
- [Instance-level quantitative saliency in multiple sclerosis lesion segmentation](https://mlsanity.com/paper/2406.09335/) (2024)
- [PRECISe : Prototype-Reservation for Explainable Classification under Imbalanced and Scarce-Data Settings](https://mlsanity.com/paper/2408.05754/) (2024)

## Abstract (arXiv)

Saliency maps have become a widely used method to make deep learning models more interpretable by providing post-hoc explanations of classifiers through identification of the most pertinent areas of the input medical image. They are increasingly being used in medical imaging to provide clinically plausible explanations for the decisions the neural network makes. However, the utility and robustness of these visualization maps has not yet been rigorously examined in the context of medical imaging. We posit that trustworthiness in this context requires 1) localization utility, 2) sensitivity to model weight randomization, 3) repeatability, and 4) reproducibility. Using the localization information available in two large public radiology datasets, we quantify the performance of eight commonly used saliency map approaches for the above criteria using area under the precision-recall curves (AUPRC) and structural similarity index (SSIM), comparing their performance to various baseline measures. Using our framework to quantify the trustworthiness of saliency maps, we show that all eight saliency map techniques fail at least one of the criteria and are, in most cases, less trustworthy when compared to the baselines. We suggest that their usage in the high-risk domain of medical imaging warrants additional scrutiny and recommend that detection or segmentation models be used if localization is the desired output of the network. Additionally, to promote reproducibility of our findings, we provide the code we used for all tests performed in this work at this link: https://github.com/QTIM-Lab/Assessing-Saliency-Maps.