# Quantitative Evaluations on Saliency Methods: An Experimental Study

arXiv: https://arxiv.org/abs/2012.15616 · HTML: https://mlsanity.com/paper/2012.15616/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 10 of 10)

- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [Connecting Interpretability and Robustness in Decision Trees through Separation](https://mlsanity.com/paper/2102.07048/) (2021)
- [Resisting Out-of-Distribution Data Problem in Perturbation of XAI](https://mlsanity.com/paper/2107.14000/) (2021)
- [Human Attention-Guided Explainable Artificial Intelligence for Computer Vision Models](https://mlsanity.com/paper/2305.03601/) (2023)
- [Saliency-Bench: A Comprehensive Benchmark for Evaluating Visual Explanations](https://mlsanity.com/paper/2310.08537/) (2023)
- [Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models](https://mlsanity.com/paper/2406.12649/) (2024)
- [Evaluation of Saliency-based Explainability Method](https://mlsanity.com/paper/2106.12773/) (2021)
- [A New Baseline Assumption of Integated Gradients Based on Shaply value](https://mlsanity.com/paper/2310.04821/) (2023)
- [Cloud-based XAI Services for Assessing Open Repository Models Under Adversarial Attacks](https://mlsanity.com/paper/2401.12261/) (2024)
- [Generating detailed saliency maps using model-agnostic methods](https://mlsanity.com/paper/2209.09202/) (2022)

## Abstract (arXiv)

It has been long debated that eXplainable AI (XAI) is an important topic, but it lacks rigorous definition and fair metrics. In this paper, we briefly summarize the status quo of the metrics, along with an exhaustive experimental study based on them, including faithfulness, localization, false-positives, sensitivity check, and stability. With the experimental results, we conclude that among all the methods we compare, no single explanation method dominates others in all metrics. Nonetheless, Gradient-weighted Class Activation Mapping (Grad-CAM) and Randomly Input Sampling for Explanation (RISE) perform fairly well in most of the metrics. Utilizing a set of filtered metrics, we further present a case study to diagnose the classification bases for models. While providing a comprehensive experimental study of metrics, we also examine measuring factors that are missed in current metrics and hope this valuable work could serve as a guide for future research.