# Neighboring Words Affect Human Interpretation of Saliency Explanations

arXiv: https://arxiv.org/abs/2305.02679 · HTML: https://mlsanity.com/paper/2305.02679/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 23 of 23)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [A Survey of the State of Explainable AI for Natural Language Processing](https://mlsanity.com/paper/2010.00711/) (2020)
- [Post-hoc Interpretability for Neural NLP: A Survey](https://mlsanity.com/paper/2108.04840/) (2021)
- [The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models](https://mlsanity.com/paper/2008.05122/) (2020)
- [Do Feature Attribution Methods Correctly Attribute Features?](https://mlsanity.com/paper/2104.14403/) (2021)
- [The effectiveness of feature attribution methods and its correlation with automatic evaluation scores](https://mlsanity.com/paper/2105.14944/) (2021)
- [Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis](https://mlsanity.com/paper/2111.04138/) (2021)
- ["What is Relevant in a Text Document?": An Interpretable Machine Learning Approach](https://mlsanity.com/paper/1612.07843/) (2016)
- [Gradient-based Analysis of NLP Models is Manipulable](https://mlsanity.com/paper/2010.05419/) (2020)
- [Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations](https://mlsanity.com/paper/2112.09669/) (2021)
- [Human Interpretation of Saliency-based Explanation Over Text](https://mlsanity.com/paper/2201.11569/) (2022)
- [On the Interaction of Belief Bias and Explanations](https://mlsanity.com/paper/2106.15355/) (2021)
- [Challenging common interpretability assumptions in feature attribution explanations](https://mlsanity.com/paper/2012.02748/) (2020)
- [Scalable visualisation methods for modern Generalized Additive Models](https://mlsanity.com/paper/1809.10632/) (2018)
- [Generating Hierarchical Explanations on Text Classification Without Connecting Rules](https://mlsanity.com/paper/2210.13270/) (2022)

## Built on by (most cited first, 4 of 4)

- [Layer-Wise Relevance Propagation with Conservation Property for ResNet](https://mlsanity.com/paper/2407.09115/) (2024)
- [Evaluating Saliency Explanations in NLP by Crowdsourcing](https://mlsanity.com/paper/2405.10767/) (2024)
- [Why is "Chicago" Predictive of Deceptive Reviews? Using LLMs to Discover Language Phenomena from Lexical Cues](https://mlsanity.com/paper/2511.13658/) (2025)
- [Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution](https://mlsanity.com/paper/2512.11108/) (2025)

## Abstract (arXiv)

Word-level saliency explanations ("heat maps over words") are often used to communicate feature-attribution in text-based models. Recent studies found that superficial factors such as word length can distort human interpretation of the communicated saliency scores. We conduct a user study to investigate how the marking of a word's neighboring words affect the explainee's perception of the word's importance in the context of a saliency explanation. We find that neighboring words have significant effects on the word's importance rating. Concretely, we identify that the influence changes based on neighboring direction (left vs. right) and a-priori linguistic and computational measures of phrases and collocations (vs. unrelated neighboring words). Our results question whether text-based saliency explanations should be continued to be communicated at word level, and inform future research on alternative saliency explanation methods.