# Investigating sanity checks for saliency maps with image and text classification

arXiv: https://arxiv.org/abs/2106.07475 · HTML: https://mlsanity.com/paper/2106.07475/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 11 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019)
- [When Explanations Lie: Why Many Modified BP Attributions Fail](https://mlsanity.com/paper/1912.09818/) (2019)

## Built on by (most cited first, 11 of 11)

- [The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus](https://mlsanity.com/paper/2302.07265/) (2023)
- [Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations](https://mlsanity.com/paper/2211.12486/) (2022)
- [ReAGent: A Model-agnostic Feature Attribution Method for Generative Language Models](https://mlsanity.com/paper/2402.00794/) (2024)
- [Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools](https://mlsanity.com/paper/2108.13961/) (2021)
- [Sanity Checks Revisited: An Exploration to Repair the Model Parameter Randomisation Test](https://mlsanity.com/paper/2401.06465/) (2024)
- [A Fresh Look at Sanity Checks for Saliency Maps](https://mlsanity.com/paper/2405.02383/) (2024)
- [Black-box language model explanation by context length probing](https://mlsanity.com/paper/2212.14815/) (2022)
- [Occlusion Sensitivity Analysis with Augmentation Subspace Perturbation in Deep Feature Space](https://mlsanity.com/paper/2311.15022/) (2023)
- [Reconciling Privacy and Explainability in High-Stakes: A Systematic Inquiry](https://mlsanity.com/paper/2412.20798/) (2024)
- [Saliency Methods are Encoders: Analysing Logical Relations Towards Interpretation](https://mlsanity.com/paper/2412.16204/) (2024)
- [Unifying Perplexing Behaviors in Modified BP Attributions through Alignment Perspective](https://mlsanity.com/paper/2503.11160/) (2025)

## Abstract (arXiv)

Saliency maps have shown to be both useful and misleading for explaining model predictions especially in the context of images. In this paper, we perform sanity checks for text modality and show that the conclusions made for image do not directly transfer to text. We also analyze the effects of the input multiplier in certain saliency maps using similarity scores, max-sensitivity and infidelity evaluation metrics. Our observations reveal that the input multiplier carries input's structural patterns in explanation maps, thus leading to similar results regardless of the choice of model parameters. We also show that the smoothness of a Neural Network (NN) function can affect the quality of saliency-based explanations. Our investigations reveal that replacing ReLUs with Softplus and MaxPool with smoother variants such as LogSumExp (LSE) can lead to explanations that are more reliable based on the infidelity evaluation metric.