# Weighted Automata Extraction and Explanation of Recurrent Neural Networks for Natural Language Tasks

arXiv: https://arxiv.org/abs/2306.14040 · HTML: https://mlsanity.com/paper/2306.14040/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 36 references that are in the map.
- First posted on arXiv 24 Jun 2023.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples](https://mlsanity.com/paper/1711.09576/) (2017)
- [Learning Deterministic Weighted Automata with Queries and Counterexamples](https://mlsanity.com/paper/1910.13895/) (2019)
- [An Empirical Evaluation of Rule Extraction from Recurrent Neural Networks](https://mlsanity.com/paper/1709.10380/) (2017)
- [Weighted Automata Extraction from Recurrent Neural Networks via Regression on State Spaces](https://mlsanity.com/paper/1904.02931/) (2019)
- [Learning with Interpretable Structure from Gated RNN](https://mlsanity.com/paper/1810.10708/) (2018)
- [Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov Models](https://mlsanity.com/paper/1606.05320/) (2016)
- [Verification of Recurrent Neural Networks Through Rule Extraction](https://mlsanity.com/paper/1811.06029/) (2018)
- [An interpretable LSTM neural network for autoregressive exogenous model](https://mlsanity.com/paper/1804.05251/) (2018)

## Built on by (most cited first, 7 of 7)

- [Boosting Jailbreak Attack with Momentum](https://mlsanity.com/paper/2405.01229/) (2024)
- [On the Duality Between Sharpness-Aware Minimization and Adversarial Training](https://mlsanity.com/paper/2402.15152/) (2024)
- [LUNA: A Model-Based Universal Analysis Framework for Large Language Models](https://mlsanity.com/paper/2310.14211/) (2023)
- [Causal Abstraction in Model Interpretability: A Compact Survey](https://mlsanity.com/paper/2410.20161/) (2024)
- [Enhancing Adversarial Attacks: The Similar Target Method](https://mlsanity.com/paper/2308.10743/) (2023)
- [Automata Extraction from Transformers](https://mlsanity.com/paper/2406.05564/) (2024)
- [ReGA: Model-Based Safeguard for LLMs via Representation-Guided Abstraction](https://mlsanity.com/paper/2506.01770/) (2025)

## Abstract (arXiv)

Recurrent Neural Networks (RNNs) have achieved tremendous success in processing sequential data, yet understanding and analyzing their behaviours remains a significant challenge. To this end, many efforts have been made to extract finite automata from RNNs, which are more amenable for analysis and explanation. However, existing approaches like exact learning and compositional approaches for model extraction have limitations in either scalability or precision. In this paper, we propose a novel framework of Weighted Finite Automata (WFA) extraction and explanation to tackle the limitations for natural language tasks. First, to address the transition sparsity and context loss problems we identified in WFA extraction for natural language tasks, we propose an empirical method to complement missing rules in the transition diagram, and adjust transition matrices to enhance the context-awareness of the WFA. We also propose two data augmentation tactics to track more dynamic behaviours of RNN, which further allows us to improve the extraction precision. Based on the extracted model, we propose an explanation method for RNNs including a word embedding method -- Transition Matrix Embeddings (TME) and TME-based task oriented explanation for the target RNN. Our evaluation demonstrates the advantage of our method in extraction precision than existing approaches, and the effectiveness of TME-based explanation method in applications to pretraining and adversarial example generation.