# Explaining and Interpreting LSTMs

arXiv: https://arxiv.org/abs/1909.12114 · HTML: https://mlsanity.com/paper/1909.12114/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2019.

## Built on by (most cited first, 23 of 23)

- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy](https://mlsanity.com/paper/2106.13200/) (2021)
- [On the Explainability of Natural Language Processing Deep Models](https://mlsanity.com/paper/2210.06929/) (2022)
- [Toward Explainable AI for Regression Models](https://mlsanity.com/paper/2112.11407/) (2021)
- [Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces](https://mlsanity.com/paper/2212.14855/) (2022)
- [Language Model Circuits Are Sparse in the Neuron Basis](https://mlsanity.com/paper/2601.22594/) (2026)
- [The Clever Hans Effect in Unsupervised Learning](https://mlsanity.com/paper/2408.08041/) (2024)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Features](https://mlsanity.com/paper/2408.17198/) (2024)
- [Recent Advances in Malware Detection: Graph Learning and Explainability](https://mlsanity.com/paper/2502.10556/) (2025)
- [Explainable Online Lane Change Predictions on a Digital Twin with a Layer Normalized LSTM and Layer-wise Relevance Propagation](https://mlsanity.com/paper/2204.01292/) (2022)
- [Implicit N-grams Induced by Recurrence](https://mlsanity.com/paper/2205.02724/) (2022)
- [Learning Temporal Rules from Noisy Timeseries Data](https://mlsanity.com/paper/2202.05403/) (2022)
- [NeuroView-RNN: It's About Time](https://mlsanity.com/paper/2202.11811/) (2022)
- [Normalized Relevance Measure as a Unifying Framework to Explain Neural Network Latent Structures](https://mlsanity.com/paper/2606.00557/) (2026)
- [Machine Learning based prediction of Glucose Levels in Type 1 Diabetes Patients with the use of Continuous Glucose Monitoring Data](https://mlsanity.com/paper/2302.12856/) (2023)
- [Detection of Dataset Shifts in Learning-Enabled Cyber-Physical Systems using Variational Autoencoder for Regression](https://mlsanity.com/paper/2104.06613/) (2021)
- [Unlocking Layer-wise Relevance Propagation for Autoencoders](https://mlsanity.com/paper/2303.11734/) (2023)
- [GAMER-MRIL identifies Disability-Related Brain Changes in Multiple Sclerosis](https://mlsanity.com/paper/2308.07611/) (2023)
- [Machine-Learned Closure of URANS for Stably Stratified Turbulence: Connecting Physical Timescales & Data Hyperparameters of Deep Time-Series Models](https://mlsanity.com/paper/2404.16141/) (2024)
- [Attribution assignment for deep-generative sequence models enables interpretability analysis using positive-only data](https://mlsanity.com/paper/2506.23182/) (2025)
- [Explaining Temporal Graph Neural Networks via Feature-induced Information Flow](https://mlsanity.com/paper/2606.27201/) (2026)
- [Explaining Hyperbolic Neural Networks via Geometry-Aware Relevance Propagation](https://mlsanity.com/paper/2609.35128/) (2026)

## Abstract (arXiv)

While neural networks have acted as a strong unifying force in the design of modern AI systems, the neural network architectures themselves remain highly heterogeneous due to the variety of tasks to be solved. In this chapter, we explore how to adapt the Layer-wise Relevance Propagation (LRP) technique used for explaining the predictions of feed-forward networks to the LSTM architecture used for sequential data modeling and forecasting. The special accumulators and gated interactions present in the LSTM require both a new propagation scheme and an extension of the underlying theoretical framework to deliver faithful explanations.