# Evaluating Recurrent Neural Network Explanations

arXiv: https://arxiv.org/abs/1904.11829 · HTML: https://mlsanity.com/paper/1904.11829/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2019.

## Built on by (most cited first, 29 of 29)

- [Do Feature Attribution Methods Correctly Attribute Features?](https://mlsanity.com/paper/2104.14403/) (2021)
- [The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations](https://mlsanity.com/paper/2106.00786/) (2021)
- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [Do Input Gradients Highlight Discriminative Features?](https://mlsanity.com/paper/2102.12781/) (2021)
- [Evaluating Saliency Methods for Neural Language Models](https://mlsanity.com/paper/2104.05824/) (2021)
- [ferret: a Framework for Benchmarking Explainers on Transformers](https://mlsanity.com/paper/2208.01575/) (2022)
- [Discovering Transferable Forensic Features for CNN-generated Images Detection](https://mlsanity.com/paper/2208.11342/) (2022)
- [A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?](https://mlsanity.com/paper/2304.04780/) (2023)
- [Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View](https://mlsanity.com/paper/2112.02625/) (2021)
- [ExSum: From Local Explanations to Model Understanding](https://mlsanity.com/paper/2205.00130/) (2022)
- [The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics](https://mlsanity.com/paper/2305.11806/) (2023)
- [On the Impact of Temporal Concept Drift on Model Explanations](https://mlsanity.com/paper/2210.09197/) (2022)
- [VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives](https://mlsanity.com/paper/2206.11212/) (2022)
- [Attention Meets Post-hoc Interpretability: A Mathematical Perspective](https://mlsanity.com/paper/2402.03485/) (2024)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [Can We Trust Explainable AI Methods on ASR? An Evaluation on Phoneme Recognition](https://mlsanity.com/paper/2305.18011/) (2023)
- [Time Series Model Attribution Visualizations as Explanations](https://mlsanity.com/paper/2109.12935/) (2021)
- [Unveiling Language Competence Neurons: A Psycholinguistic Approach to Model Interpretability](https://mlsanity.com/paper/2409.15827/) (2024)
- [Schr"odinger's Tree -- On Syntax and Neural Language Models](https://mlsanity.com/paper/2110.08887/) (2021)
- [Threading the Needle of On and Off-Manifold Value Functions for Shapley Explanations](https://mlsanity.com/paper/2202.11919/) (2022)
- [Applications of Explainable AI for 6G: Technical Aspects, Use Cases, and Research Challenges](https://mlsanity.com/paper/2112.04698/) (2021)
- [ReX: A Framework for Incorporating Temporal Information in Model-Agnostic Local Explanation Techniques](https://mlsanity.com/paper/2209.03798/) (2022)
- [B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainability](https://mlsanity.com/paper/2502.12992/) (2025)
- [Do End-to-End Speech Recognition Models Care About Context?](https://mlsanity.com/paper/2102.09928/) (2021)
- [Improving Zero-Shot Cross-Lingual Transfer via Progressive Code-Switching](https://mlsanity.com/paper/2406.13361/) (2024)
- [More Than Words: Towards Better Quality Interpretations of Text Classifiers](https://mlsanity.com/paper/2112.12444/) (2021)
- [Insights Into the Inner Workings of Transformer Models for Protein Function Prediction](https://mlsanity.com/paper/2309.03631/) (2023)
- [Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation](https://mlsanity.com/paper/2109.06308/) (2021)
- [Bridging Fairness and Explainability: Can Input-Based Explanations Promote Fairness in Hate Speech Detection?](https://mlsanity.com/paper/2509.22291/) (2025)

## Abstract (arXiv)

Recently, several methods have been proposed to explain the predictions of recurrent neural networks (RNNs), in particular of LSTMs. The goal of these methods is to understand the network's decisions by assigning to each input variable, e.g., a word, a relevance indicating to which extent it contributed to a particular prediction. In previous works, some of these methods were not yet compared to one another, or were evaluated only qualitatively. We close this gap by systematically and quantitatively comparing these methods in different settings, namely (1) a toy arithmetic task which we use as a sanity check, (2) a five-class sentiment prediction of movie reviews, and besides (3) we explore the usefulness of word relevances to build sentence-level representations. Lastly, using the method that performed best in our experiments, we show how specific linguistic phenomena such as the negation in sentiment analysis reflect in terms of relevance patterns, and how the relevance visualization can help to understand the misclassification of individual samples.