# Learning Finite State Representations of Recurrent Policy Networks

arXiv: https://arxiv.org/abs/1811.12530 · HTML: https://mlsanity.com/paper/1811.12530/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 29 Nov 2018.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Built on by (most cited first, 30 of 37)

- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning](https://mlsanity.com/paper/2101.12446/) (2021)
- [Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods](https://mlsanity.com/paper/2102.13045/) (2021)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Classical Planning in Deep Latent Space](https://mlsanity.com/paper/2107.00110/) (2021)
- [Explainable Reinforcement Learning via a Causal World Model](https://mlsanity.com/paper/2305.02749/) (2023)
- [SoK: Explainable Machine Learning for Computer Security Applications](https://mlsanity.com/paper/2208.10605/) (2022)
- [Explainable Autonomous Robots: A Survey and Perspective](https://mlsanity.com/paper/2105.02658/) (2021)
- [Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search](https://mlsanity.com/paper/2405.16450/) (2024)
- [Mo"ET: Mixture of Expert Trees and its Application to Verifiable Reinforcement Learning](https://mlsanity.com/paper/1906.06717/) (2019)
- [Explaining RL Decisions with Trajectories](https://mlsanity.com/paper/2305.04073/) (2023)
- [Pessimistic Iterative Planning with RNNs for Robust POMDPs](https://mlsanity.com/paper/2408.08770/) (2024)
- [Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations](https://mlsanity.com/paper/2502.12530/) (2025)
- [Local Explanations for Reinforcement Learning](https://mlsanity.com/paper/2202.03597/) (2022)
- [Generating Probabilistic Safety Guarantees for Neural Network Controllers](https://mlsanity.com/paper/2103.01203/) (2021)
- [Learning Symbolic Representations for Reinforcement Learning of Non-Markovian Behavior](https://mlsanity.com/paper/2301.02952/) (2023)
- [Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU Networks](https://mlsanity.com/paper/2308.02729/) (2023)
- [Learning-Aided Heuristics Design for Storage System](https://mlsanity.com/paper/2106.07288/) (2021)
- [Identifying Reasoning Flaws in Planning-Based RL Using Tree Explanations](https://mlsanity.com/paper/2109.13978/) (2021)
- [A Framework for Understanding and Visualizing Strategies of RL Agents](https://mlsanity.com/paper/2208.08552/) (2022)
- [Knowledge-based XAI through CBR: There is more to explanations than models can tell](https://mlsanity.com/paper/2108.10363/) (2021)
- [Program Machine Policy: Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machines](https://mlsanity.com/paper/2311.15960/) (2023)
- [Training Neural Networks with Internal State, Unconstrained Connectivity, and Discrete Activations](https://mlsanity.com/paper/2312.14359/) (2023)
- [Finite-State Controllers for (Hidden-Model) POMDPs using Deep Reinforcement Learning](https://mlsanity.com/paper/2602.08734/) (2026)
- [Longitudinal Distance: Towards Accountable Instance Attribution](https://mlsanity.com/paper/2108.10437/) (2021)
- [Beyond Value: CHECKLIST for Testing Inferences in Planning-Based RL](https://mlsanity.com/paper/2206.02039/) (2022)
- [Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learning](https://mlsanity.com/paper/2211.06376/) (2022)
- [IxDRL: A Novel Explainable Deep Reinforcement Learning Toolkit based on Analyses of Interestingness](https://mlsanity.com/paper/2307.08933/) (2023)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features. The result is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior. We present results of this approach on synthetic environments and six Atari games. The resulting finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy. We also show that these finite policy representations lead to improved interpretability.