# Metrics for saliency map evaluation of deep learning explanation methods

arXiv: https://arxiv.org/abs/2201.13291 · HTML: https://mlsanity.com/paper/2201.13291/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Neural Prototype Trees for Interpretable Fine-grained Image Recognition](https://mlsanity.com/paper/2012.02046/) (2020)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Reliable Post hoc Explanations: Modeling Uncertainty in Explainability](https://mlsanity.com/paper/2008.05030/) (2020)
- [Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study](https://mlsanity.com/paper/2002.00772/) (2020)
- [Interpretable and Accurate Fine-grained Recognition via Region Grouping](https://mlsanity.com/paper/2005.10411/) (2020)
- [See Better Before Looking Closer: Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classification](https://mlsanity.com/paper/1901.09891/) (2019)
- [Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values](https://mlsanity.com/paper/1810.03307/) (2018)

## Built on by (most cited first, 30 of 40)

- [LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity](https://mlsanity.com/paper/2404.03214/) (2024)
- [Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks](https://mlsanity.com/paper/2401.06654/) (2024)
- [Opti-CAM: Optimizing saliency maps for interpretability](https://mlsanity.com/paper/2301.07002/) (2023)
- [A comprehensive study on fidelity metrics for XAI](https://mlsanity.com/paper/2401.10640/) (2024)
- [Deletion and Insertion Tests in Regression Models](https://mlsanity.com/paper/2205.12423/) (2022)
- [Towards Better Explanations for Object Detection](https://mlsanity.com/paper/2306.02744/) (2023)
- [Saliency Cards: A Framework to Characterize and Compare Saliency Methods](https://mlsanity.com/paper/2206.02958/) (2022)
- [How explainable are adversarially-robust CNNs?](https://mlsanity.com/paper/2205.13042/) (2022)
- [Sanity checks for patch visualisation in prototype-based image classification](https://mlsanity.com/paper/2311.16120/) (2023)
- [An adversarial attack approach for eXplainable AI evaluation on deepfake detection models](https://mlsanity.com/paper/2312.06627/) (2023)
- [Improving the Evaluation and Actionability of Explanation Methods for Multivariate Time Series Classification](https://mlsanity.com/paper/2406.12507/) (2024)
- [A novel approach to generate datasets with XAI ground truth to evaluate image models](https://mlsanity.com/paper/2302.05624/) (2023)
- [Reliable Evaluation of Attribution Maps in CNNs: A Perturbation-Based Approach](https://mlsanity.com/paper/2411.14946/) (2024)
- [Are explainable AI (XAI) evaluation strategies aligned? Comparing subjective, objective, and mathematical evaluation measures using saliency maps](https://mlsanity.com/paper/2504.17023/) (2025)
- [Sanity checks and improvements for patch visualisation in prototype-based image classification](https://mlsanity.com/paper/2302.08508/) (2023)
- [Explanation of Face Recognition via Saliency Maps](https://mlsanity.com/paper/2304.06118/) (2023)
- [IBO: Inpainting-Based Occlusion to Enhance Explainable Artificial Intelligence Evaluation in Histopathology](https://mlsanity.com/paper/2408.16395/) (2024)
- [ODExAI: A Comprehensive Object Detection Explainable AI Evaluation](https://mlsanity.com/paper/2504.19249/) (2025)
- [Evaluation of Explanation Methods of AI -- CNNs in Image Classification Tasks with Reference-based and No-reference Metrics](https://mlsanity.com/paper/2212.01222/) (2022)
- [A Brief Review of Explainable Artificial Intelligence in Healthcare](https://mlsanity.com/paper/2304.01543/) (2023)
- [SPADE: Sparsity-Guided Debugging for Deep Neural Networks](https://mlsanity.com/paper/2310.04519/) (2023)
- [On the stability, correctness and plausibility of visual explanation methods based on feature importance](https://mlsanity.com/paper/2311.12860/) (2023)
- [Challenging the Black Box: A Comprehensive Evaluation of Attribution Maps of CNN Applications in Agriculture and Forestry](https://mlsanity.com/paper/2402.11670/) (2024)
- [CA-Stream: Attention-based pooling for interpretable image recognition](https://mlsanity.com/paper/2404.14996/) (2024)
- [Improving the Perturbation-Based Explanation of Deepfake Detectors Through the Use of Adversarially-Generated Samples](https://mlsanity.com/paper/2502.03957/) (2025)
- [Comparison of attention models and post-hoc explanation methods for embryo stage identification: a case study](https://mlsanity.com/paper/2205.06546/) (2022)
- [FM-G-CAM: A Holistic Approach for Explainable AI in Computer Vision](https://mlsanity.com/paper/2312.05975/) (2023)
- [Exploring the Plausibility of Hate and Counter Speech Detectors with Explainable AI](https://mlsanity.com/paper/2407.20274/) (2024)
- [Enhancing Post-Hoc Explanation Benchmark Reliability for Image Classification](https://mlsanity.com/paper/2311.17876/) (2023)
- [Explainable AI in Time-Sensitive Scenarios: Prefetched Offline Explanation Model](https://mlsanity.com/paper/2503.04283/) (2025)

## Abstract (arXiv)

Due to the black-box nature of deep learning models, there is a recent development of solutions for visual explanations of CNNs. Given the high cost of user studies, metrics are necessary to compare and evaluate these different methods. In this paper, we critically analyze the Deletion Area Under Curve (DAUC) and Insertion Area Under Curve (IAUC) metrics proposed by Petsiuk et al. (2018). These metrics were designed to evaluate the faithfulness of saliency maps generated by generic methods such as Grad-CAM or RISE. First, we show that the actual saliency score values given by the saliency map are ignored as only the ranking of the scores is taken into account. This shows that these metrics are insufficient by themselves, as the visual appearance of a saliency map can change significantly without the ranking of the scores being modified. Secondly, we argue that during the computation of DAUC and IAUC, the model is presented with images that are out of the training distribution which might lead to an unreliable behavior of the model being explained. To complement DAUC/IAUC, we propose new metrics that quantify the sparsity and the calibration of explanation methods, two previously unstudied properties. Finally, we give general remarks about the metrics studied in this paper and discuss how to evaluate them in a user study.