# Re-understanding Finite-State Representations of Recurrent Policy Networks

arXiv: https://arxiv.org/abs/2006.03745 · HTML: https://mlsanity.com/paper/2006.03745/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2020.

## Built on by (most cited first, 15 of 15)

- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations](https://mlsanity.com/paper/2302.12689/) (2023)
- [LEADER: Learning Attention over Driving Behaviors for Planning under Uncertainty](https://mlsanity.com/paper/2209.11422/) (2022)
- [Explainable Multi-Agent Reinforcement Learning for Temporal Queries](https://mlsanity.com/paper/2305.10378/) (2023)
- [Explaining RL Decisions with Trajectories](https://mlsanity.com/paper/2305.04073/) (2023)
- [Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations](https://mlsanity.com/paper/2502.12530/) (2025)
- [Local Explanations for Reinforcement Learning](https://mlsanity.com/paper/2202.03597/) (2022)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents](https://mlsanity.com/paper/2101.07312/) (2021)
- [Learning Probabilistic Temporal Logic Specifications for Stochastic Systems](https://mlsanity.com/paper/2505.12107/) (2025)
- [Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning](https://mlsanity.com/paper/2307.05209/) (2023)
- [Know your Trajectory -- Trustworthy Reinforcement Learning deployment through Importance-Based Trajectory Analysis](https://mlsanity.com/paper/2512.06917/) (2025)
- [A Formal Framework for the Explanation of Finite Automata Decisions](https://mlsanity.com/paper/2602.13351/) (2026)
- [Principal Prototype Analysis on Manifold for Interpretable Reinforcement Learning](https://mlsanity.com/paper/2603.27971/) (2026)
- [ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies](https://mlsanity.com/paper/2607.07235/) (2026)

## Abstract (arXiv)

We introduce an approach for understanding control policies represented as recurrent neural networks. Recent work has approached this problem by transforming such recurrent policy networks into finite-state machines (FSM) and then analyzing the equivalent minimized FSM. While this led to interesting insights, the minimization process can obscure a deeper understanding of a machine's operation by merging states that are semantically distinct. To address this issue, we introduce an analysis approach that starts with an unminimized FSM and applies more-interpretable reductions that preserve the key decision points of the policy. We also contribute an attention tool to attain a deeper understanding of the role of observations in the decisions. Our case studies on 7 Atari games and 3 control benchmarks demonstrate that the approach can reveal insights that have not been previously noticed.