mlsanityData to 14 Oct 2026

Learning Finite State Representations of Recurrent Policy Networks

arXiv 1811.12530 ↗First posted 29 Nov 2018cs.LG stat.MLcited by 37

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features. The result is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior. We present results of this approach on synthetic environments and six Atari games. The resulting finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy. We also show that these finite policy representations lead to improved interpretability.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 29 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 11; 2022: 6; 2023: 8; 2024: 3; 2025: 2; 2026: 6.

37 papers.

Sort:

By area:

  1. 2021Acquisition of Chess Knowledge in AlphaZerocited by 123
  2. 2021A Survey on Interpretable Reinforcement Learningcited by 73
  3. 2023Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZerocited by 42
  4. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 39
  5. 2021Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methodscited by 31
  6. 2022A Survey of Explainable Reinforcement Learningcited by 31
  7. 2021Classical Planning in Deep Latent Spacecited by 22
  8. 2023Explainable Reinforcement Learning via a Causal World Modelcited by 22
  9. 2022SoK: Explainable Machine Learning for Computer Security Applicationscited by 20
  10. 2021Explainable Autonomous Robots: A Survey and Perspectivecited by 15
  11. 2024Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Searchcited by 13
  12. 2019Mo"ET: Mixture of Expert Trees and its Application to Verifiable Reinforcement Learningcited by 12
25 more
  1. 2023Explaining RL Decisions with Trajectoriescited by 7
  2. 2024Pessimistic Iterative Planning with RNNs for Robust POMDPscited by 7
  3. 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
  4. 2022Local Explanations for Reinforcement Learningcited by 5
  5. 2021Generating Probabilistic Safety Guarantees for Neural Network Controllerscited by 4
  6. 2023Learning Symbolic Representations for Reinforcement Learning of Non-Markovian Behaviorcited by 4
  7. 2023Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU Networkscited by 3
  8. 2021Learning-Aided Heuristics Design for Storage Systemcited by 2
  9. 2021Identifying Reasoning Flaws in Planning-Based RL Using Tree Explanationscited by 2
  10. 2022A Framework for Understanding and Visualizing Strategies of RL Agentscited by 2
  11. 2021Knowledge-based XAI through CBR: There is more to explanations than models can tellcited by 1
  12. 2023Program Machine Policy: Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machinescited by 1
  13. 2023Training Neural Networks with Internal State, Unconstrained Connectivity, and Discrete Activationscited by 1
  14. 2026Finite-State Controllers for (Hidden-Model) POMDPs using Deep Reinforcement Learningcited by 1
  15. 2021Longitudinal Distance: Towards Accountable Instance Attributioncited by 0
  16. 2022Beyond Value: CHECKLIST for Testing Inferences in Planning-Based RLcited by 0
  17. 2022Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learningcited by 0
  18. 2023IxDRL: A Novel Explainable Deep Reinforcement Learning Toolkit based on Analyses of Interestingnesscited by 0
  19. 2024Explainable Representation of Finite-Memory Policies for POMDPs using Decision Treescited by 0
  20. 2025Efficiently Vectorized MCMC on Modern Acceleratorscited by 0
  21. 2026Principal Prototype Analysis on Manifold for Interpretable Reinforcement Learningcited by 0
  22. 2026Compositional Behavioral Semantics for State Abstraction in Reinforcement Learningcited by 0
  23. 2026ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policiescited by 0
  24. 2026Explaining Reinforcement Learning Agents via Inductive Logic Programmingcited by 0
  25. 2026From Black Box to Executable Logic: Explainable Reinforcement Learning through Prolog Expert Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning
  2. 2026Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learning
  3. 2025A Survey of State Representation Learning for Deep Reinforcement Learning
  4. 2026Probabilistic Verification of Recurrent Neural Networks for Single and Multi-Agent Reinforcement Learning
  5. 2026Reinforcement Learning with Complex (valued) Memories
  6. 2024Bridging State and History Representations: Understanding Self-Predictive RL
  7. 2026Does Learning Recurrent Networks Matter in Partially Observable RL?
  8. 2015On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models