# Automatic Rule Extraction from Long Short Term Memory Networks

arXiv: https://arxiv.org/abs/1702.02540 · HTML: https://mlsanity.com/paper/1702.02540/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2017.

## Built on by (most cited first, 10 of 10)

- [DEGREE: Decomposition Based Explanation For Graph Neural Networks](https://mlsanity.com/paper/2305.12895/) (2023)
- [Explainable Bayesian Optimization](https://mlsanity.com/paper/2401.13334/) (2024)
- [Implicit N-grams Induced by Recurrence](https://mlsanity.com/paper/2205.02724/) (2022)
- [Pairing Conceptual Modeling with Machine Learning](https://mlsanity.com/paper/2106.14251/) (2021)
- [State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions](https://mlsanity.com/paper/2212.05178/) (2022)
- [Explainability-aided Domain Generalization for Image Classification](https://mlsanity.com/paper/2104.01742/) (2021)
- [Explaining the Deep Natural Language Processing by Mining Textual Interpretable Features](https://mlsanity.com/paper/2106.06697/) (2021)
- [Interpreting and improving deep-learning models with reality checks](https://mlsanity.com/paper/2108.06847/) (2021)
- [Explainability of Text Processing and Retrieval Methods: A Survey](https://mlsanity.com/paper/2212.07126/) (2022)
- [Posterior-driven Heuristic Support Adaptation in a Probabilistic Treatment of Real2Sim2Real for Vision-Driven Deformable Linear Object Manipulation](https://mlsanity.com/paper/2510.26656/) (2025)

## Abstract (arXiv)

Although deep learning models have proven effective at solving problems in natural language processing, the mechanism by which they come to their conclusions is often unclear. As a result, these models are generally treated as black boxes, yielding no insight of the underlying learned patterns. In this paper we consider Long Short Term Memory networks (LSTMs) and demonstrate a new approach for tracking the importance of a given input to the LSTM for a given output. By identifying consistently important patterns of words, we are able to distill state of the art LSTMs on sentiment analysis and question answering into a set of representative phrases. This representation is then quantitatively validated by using the extracted phrases to construct a simple, rule-based classifier which approximates the output of the LSTM.