# Synthesizing Context-free Grammars from Recurrent Neural Networks (Extended Version)

arXiv: https://arxiv.org/abs/2101.08200 · HTML: https://mlsanity.com/paper/2101.08200/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 20 Jan 2021.

## Builds on (most cited first, 9 of 9)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples](https://mlsanity.com/paper/1711.09576/) (2017)
- [RNNs can generate bounded hierarchical languages with optimal memory](https://mlsanity.com/paper/2010.07515/) (2020)
- [On the Computational Power of RNNs](https://mlsanity.com/paper/1906.06349/) (2019)
- [Rule Extraction Algorithm for Deep Neural Networks: A Review](https://mlsanity.com/paper/1610.05267/) (2016)
- [Evaluating the Ability of LSTMs to Learn Context-Free Grammars](https://mlsanity.com/paper/1811.02611/) (2018)
- [Explaining Black Boxes on Sequential Data using Weighted Automata](https://mlsanity.com/paper/1810.05741/) (2018)
- [Connecting First and Second Order Recurrent Networks with Deterministic Finite Automata](https://mlsanity.com/paper/1911.04644/) (2019)

## Built on by (most cited first, 2 of 2)

- [Anomaly Detection with Neural Parsers That Never Reject](https://mlsanity.com/paper/2108.00103/) (2021)
- [Unsupervised and Few-shot Parsing from Pretrained Language Models](https://mlsanity.com/paper/2206.04980/) (2022)

## Abstract (arXiv)

We present an algorithm for extracting a subclass of the context free grammars (CFGs) from a trained recurrent neural network (RNN). We develop a new framework, pattern rule sets (PRSs), which describe sequences of deterministic finite automata (DFAs) that approximate a non-regular language. We present an algorithm for recovering the PRS behind a sequence of such automata, and apply it to the sequences of automata extracted from trained RNNs using the L* algorithm. We then show how the PRS may converted into a CFG, enabling a familiar and useful presentation of the learned language. Extracting the learned language of an RNN is important to facilitate understanding of the RNN and to verify its correctness. Furthermore, the extracted CFG can augment the RNN in classifying correct sentences, as the RNN's predictive accuracy decreases when the recursion depth and distance between matching delimiters of its input sequences increases.