# Evaluating Input Perturbation Methods for Interpreting CNNs and Saliency Map Comparison

arXiv: https://arxiv.org/abs/2101.10977 · HTML: https://mlsanity.com/paper/2101.10977/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 26 Jan 2021.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Sanity Checks for Saliency Metrics](https://mlsanity.com/paper/1912.01451/) (2019)
- [Towards better understanding of gradient-based attribution methods for Deep Neural Networks](https://mlsanity.com/paper/1711.06104/) (2017)

## Built on by (most cited first, 7 of 7)

- [The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus](https://mlsanity.com/paper/2302.07265/) (2023)
- [New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound](https://mlsanity.com/paper/2211.02912/) (2022)
- [Believe The HiPe: Hierarchical Perturbation for Fast, Robust, and Model-Agnostic Saliency Mapping](https://mlsanity.com/paper/2103.05108/) (2021)
- [On the Evaluation Consistency of Attribution-based Explanations](https://mlsanity.com/paper/2407.19471/) (2024)
- [From Flexibility to Manipulation: The Slippery Slope of XAI Evaluation](https://mlsanity.com/paper/2412.05592/) (2024)
- [BOREx: Bayesian-Optimization--Based Refinement of Saliency Map for Image- and Video-Classification Models](https://mlsanity.com/paper/2210.17130/) (2022)
- [Evaluate with the Inverse: Efficient Approximation of Latent Explanation Quality Distribution](https://mlsanity.com/paper/2502.15403/) (2025)

## Abstract (arXiv)

Input perturbation methods occlude parts of an input to a function and measure the change in the function's output. Recently, input perturbation methods have been applied to generate and evaluate saliency maps from convolutional neural networks. In practice, neutral baseline images are used for the occlusion, such that the baseline image's impact on the classification probability is minimal. However, in this paper we show that arguably neutral baseline images still impact the generated saliency maps and their evaluation with input perturbations. We also demonstrate that many choices of hyperparameters lead to the divergence of saliency maps generated by input perturbations. We experimentally reveal inconsistencies among a selection of input perturbation methods and find that they lack robustness for generating saliency maps and for evaluating saliency maps as saliency metrics.