# An Empirical Evaluation of Rule Extraction from Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1709.10380 · HTML: https://mlsanity.com/paper/1709.10380/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2017.

## Built on by (most cited first, 21 of 21)

- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [Learning Transformer Programs](https://mlsanity.com/paper/2306.01128/) (2023)
- [Minimum Description Length Recurrent Neural Networks](https://mlsanity.com/paper/2111.00600/) (2021)
- [Extracting Finite Automata from RNNs Using State Merging](https://mlsanity.com/paper/2201.12451/) (2022)
- [Extracting Weighted Finite Automata from Recurrent Neural Networks for Natural Languages](https://mlsanity.com/paper/2206.14621/) (2022)
- [DeepDFA: Automata Learning through Neural Probabilistic Relaxations](https://mlsanity.com/paper/2408.08622/) (2024)
- [Linguistically inspired roadmap for building biologically reliable protein language models](https://mlsanity.com/paper/2207.00982/) (2022)
- [Weighted Automata Extraction and Explanation of Recurrent Neural Networks for Natural Language Tasks](https://mlsanity.com/paper/2306.14040/) (2023)
- [Neurosymbolic AI and its Taxonomy: a survey](https://mlsanity.com/paper/2305.08876/) (2023)
- [ReX: A Framework for Incorporating Temporal Information in Model-Agnostic Local Explanation Techniques](https://mlsanity.com/paper/2209.03798/) (2022)
- [Extracting Weighted Automata for Approximate Minimization in Language Modelling](https://mlsanity.com/paper/2106.02965/) (2021)
- [Self-Supervised Learning to Prove Equivalence Between Straight-Line Programs via Rewrite Rules](https://mlsanity.com/paper/2109.10476/) (2021)
- [Stability Analysis of Various Symbolic Rule Extraction Methods from Recurrent Neural Network](https://mlsanity.com/paper/2402.02627/) (2024)
- [Proving Equivalence Between Complex Expressions Using Graph-to-Sequence Neural Models](https://mlsanity.com/paper/2106.02452/) (2021)
- [State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions](https://mlsanity.com/paper/2212.05178/) (2022)
- [Learning Deterministic Finite Automata from Confidence Oracles](https://mlsanity.com/paper/2311.10963/) (2023)
- [Analyzing constrained LLM through PDFA-learning](https://mlsanity.com/paper/2406.08269/) (2024)
- [Extracting Robust Register Automata from Neural Networks over Data Sequences](https://mlsanity.com/paper/2511.19100/) (2025)
- [On the Relationship Between RNN Hidden State Vectors and Semantic Ground Truth](https://mlsanity.com/paper/2306.16854/) (2023)
- [Learning Model Successors](https://mlsanity.com/paper/2502.00197/) (2025)
- [NeuroRule: Making Black-Box Neural Networks Explainable through Rule-set Evolution](https://mlsanity.com/paper/2609.26841/) (2026)

## Abstract (arXiv)

Rule extraction from black-box models is critical in domains that require model validation before implementation, as can be the case in credit scoring and medical diagnosis. Though already a challenging problem in statistical learning in general, the difficulty is even greater when highly non-linear, recursive models, such as recurrent neural networks (RNNs), are fit to data. Here, we study the extraction of rules from second-order recurrent neural networks trained to recognize the Tomita grammars. We show that production rules can be stably extracted from trained RNNs and that in certain cases the rules outperform the trained RNNs.