# Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methods

arXiv: https://arxiv.org/abs/2210.07222 · HTML: https://mlsanity.com/paper/2210.07222/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 81 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 46)

- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI](https://mlsanity.com/paper/2010.07487/) (2020)
- [Measuring Association Between Labels and Free-Text Rationales](https://mlsanity.com/paper/2010.12762/) (2020)
- [Post-hoc Interpretability for Neural NLP: A Survey](https://mlsanity.com/paper/2108.04840/) (2021)
- [Reframing Human-AI Collaboration for Generating Free-Text Explanations](https://mlsanity.com/paper/2112.08674/) (2021)
- [Learning to Faithfully Rationalize by Construction](https://mlsanity.com/paper/2005.00115/) (2020)
- [What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods](https://mlsanity.com/paper/2112.04417/) (2021)
- [Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?](https://mlsanity.com/paper/2010.04119/) (2020)
- [Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach](https://mlsanity.com/paper/2002.01092/) (2020)
- [Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions](https://mlsanity.com/paper/1901.03729/) (2019)
- [AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models](https://mlsanity.com/paper/1909.09251/) (2019)
- [Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations](https://mlsanity.com/paper/1910.03065/) (2019)
- [Improving Deep Learning Interpretability by Saliency Guided Training](https://mlsanity.com/paper/2111.14338/) (2021)
- [Training Classifiers with Natural Language Explanations](https://mlsanity.com/paper/1805.03818/) (2018)
- [Do Explanations make VQA Models more Predictable to a Human?](https://mlsanity.com/paper/1810.12366/) (2018)
- [Prompting Contrastive Explanations for Commonsense Reasoning Tasks](https://mlsanity.com/paper/2106.06823/) (2021)
- [Do Human Rationales Improve Machine Explanations?](https://mlsanity.com/paper/1905.13714/) (2019)
- [On the Diversity and Limits of Human Explanations](https://mlsanity.com/paper/2106.11988/) (2021)
- [Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision](https://mlsanity.com/paper/2012.08012/) (2020)

## Built on by (most cited first, 13 of 13)

- [XAI for All: Can Large Language Models Simplify Explainable AI?](https://mlsanity.com/paper/2401.13110/) (2024)
- [xTower: A Multilingual LLM for Explaining and Correcting Translation Errors](https://mlsanity.com/paper/2406.19482/) (2024)
- [InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanations](https://mlsanity.com/paper/2310.05592/) (2023)
- [Talking Nonsense: Probing Large Language Models' Understanding of Adversarial Gibberish Inputs](https://mlsanity.com/paper/2404.17120/) (2024)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [GraphNarrator: Generating Textual Explanations for Graph Neural Networks](https://mlsanity.com/paper/2410.15268/) (2024)
- [SIDU-TXT: An XAI Algorithm for NLP with a Holistic Assessment Approach](https://mlsanity.com/paper/2402.03043/) (2024)
- [Noiser: Bounded Input Perturbations for Attributing Large Language Models](https://mlsanity.com/paper/2504.02911/) (2025)
- [This looks like what? Challenges and Future Research Directions for Part-Prototype Models](https://mlsanity.com/paper/2502.09340/) (2025)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)
- [EvalxNLP: A Framework for Benchmarking Post-Hoc Explainability Methods on NLP Models](https://mlsanity.com/paper/2505.01238/) (2025)
- [Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales](https://mlsanity.com/paper/2606.01148/) (2026)
- [Part-based Quantitative Analysis for Heatmaps](https://mlsanity.com/paper/2405.13264/) (2024)

## Abstract (arXiv)

Saliency maps can explain a neural model's predictions by identifying important input features. They are difficult to interpret for laypeople, especially for instances with many features. In order to make them more accessible, we formalize the underexplored task of translating saliency maps into natural language and compare methods that address two key challenges of this approach -- what and how to verbalize. In both automatic and human evaluation setups, using token-level attributions from text classification tasks, we compare two novel methods (search-based and instruction-based verbalizations) against conventional feature importance representations (heatmap visualizations and extractive rationales), measuring simulatability, faithfulness, helpfulness and ease of understanding. Instructing GPT-3.5 to generate saliency map verbalizations yields plausible explanations which include associations, abstractive summarization and commonsense reasoning, achieving by far the highest human ratings, but they are not faithfully capturing numeric information and are inconsistent in their interpretation of the task. In comparison, our search-based, model-free verbalization approach efficiently completes templated verbalizations, is faithful by design, but falls short in helpfulness and simulatability. Our results suggest that saliency map verbalization makes feature attribution explanations more comprehensible and less cognitively challenging to humans than conventional representations.