# Explaining Black Boxes on Sequential Data using Weighted Automata

arXiv: https://arxiv.org/abs/1810.05741 · HTML: https://mlsanity.com/paper/1810.05741/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2018.

## Built on by (most cited first, 11 of 11)

- [Thinking Like Transformers](https://mlsanity.com/paper/2106.06981/) (2021)
- [ReX: A Framework for Incorporating Temporal Information in Model-Agnostic Local Explanation Techniques](https://mlsanity.com/paper/2209.03798/) (2022)
- [LUNA: A Model-Based Universal Analysis Framework for Large Language Models](https://mlsanity.com/paper/2310.14211/) (2023)
- [Simulating Weighted Automata over Sequences and Trees with Transformers](https://mlsanity.com/paper/2403.09728/) (2024)
- [Extracting Weighted Automata for Approximate Minimization in Language Modelling](https://mlsanity.com/paper/2106.02965/) (2021)
- [Synthesizing Context-free Grammars from Recurrent Neural Networks (Extended Version)](https://mlsanity.com/paper/2101.08200/) (2021)
- [FlexFringe: Modeling Software Behavior by Learning Probabilistic Automata](https://mlsanity.com/paper/2203.16331/) (2022)
- [Verifying And Interpreting Neural Networks using Finite Automata](https://mlsanity.com/paper/2211.01022/) (2022)
- [Learning Transductions and Alignments with RNN Seq2seq Models](https://mlsanity.com/paper/2303.06841/) (2023)
- [DeepCover: Advancing RNN Test Coverage and Online Error Prediction using State Machine Extraction](https://mlsanity.com/paper/2402.06966/) (2024)
- [Learning Weighted Finite Automata over the Max-Plus Semiring and its Termination](https://mlsanity.com/paper/2407.09775/) (2024)

## Abstract (arXiv)

Understanding how a learned black box works is of crucial interest for the future of Machine Learning. In this paper, we pioneer the question of the global interpretability of learned black box models that assign numerical values to symbolic sequential data. To tackle that task, we propose a spectral algorithm for the extraction of weighted automata (WA) from such black boxes. This algorithm does not require the access to a dataset or to the inner representation of the black box: the inferred model can be obtained solely by querying the black box, feeding it with inputs and analyzing its outputs. Experiments using Recurrent Neural Networks (RNN) trained on a wide collection of 48 synthetic datasets and 2 real datasets show that the obtained approximation is of great quality.