Re-understanding Finite-State Representations of Recurrent Policy Networks
Not on the map. Why?
Abstract (arXiv)
We introduce an approach for understanding control policies represented as recurrent neural networks. Recent work has approached this problem by transforming such recurrent policy networks into finite-state machines (FSM) and then analyzing the equivalent minimized FSM. While this led to interesting insights, the minimization process can obscure a deeper understanding of a machine's operation by merging states that are semantically distinct. To address this issue, we introduce an analysis approach that starts with an unminimized FSM and applies more-interpretable reductions that preserve the key decision points of the policy. We also contribute an attention tool to attain a deeper understanding of the role of observations in the decisions. Our case studies on 7 Atari games and 3 control benchmarks demonstrate that the approach can reveal insights that have not been previously noticed.
Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jun 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 1; 2022: 3; 2023: 5; 2024: 0; 2025: 3; 2026: 3.
15 papers.
Sort:
By area:
- 2022A Survey of Explainable Reinforcement Learningcited by 31
- 2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanationscited by 17
- 2022LEADER: Learning Attention over Driving Behaviors for Planning under Uncertaintycited by 9
- 2023Explainable Multi-Agent Reinforcement Learning for Temporal Queriescited by 9
- 2023Explaining RL Decisions with Trajectoriescited by 7
- 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
- 2022Local Explanations for Reinforcement Learningcited by 5
- 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
- 2021Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agentscited by 4
- 2025Learning Probabilistic Temporal Logic Specifications for Stochastic Systemscited by 2
- 2023Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learningcited by 1
- 2025Know your Trajectory -- Trustworthy Reinforcement Learning deployment through Importance-Based Trajectory Analysiscited by 0
- 2026ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies8 Jul 2026
- 2026Principal Prototype Analysis on Manifold for Interpretable Reinforcement Learning30 Mar 2026
- 2026A Formal Framework for the Explanation of Finite Automata Decisions12 Feb 2026
- 2025Know your Trajectory -- Trustworthy Reinforcement Learning deployment through Importance-Based Trajectory Analysis7 Dec 2025
- 2025Learning Probabilistic Temporal Logic Specifications for Stochastic Systems17 May 2025
- 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanations18 Feb 2025
- 2023Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning11 Jul 2023
- 2023Explainable Multi-Agent Reinforcement Learning for Temporal Queries17 May 2023
- 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges17 May 2023
- 2023Explaining RL Decisions with Trajectories6 May 2023
- 2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations24 Feb 2023
- 2022LEADER: Learning Attention over Driving Behaviors for Planning under Uncertainty23 Sep 2022
3 more
- 2022A Survey of Explainable Reinforcement Learning17 Feb 2022
- 2022Local Explanations for Reinforcement Learning8 Feb 2022
- 2021Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents18 Jan 2021
- 2022A Survey of Explainable Reinforcement Learningcited by 31
- 2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanationscited by 17
- 2023Explainable Multi-Agent Reinforcement Learning for Temporal Queriescited by 9
- 2023Explaining RL Decisions with Trajectoriescited by 7
- 2022Local Explanations for Reinforcement Learningcited by 5
- 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
- 2021Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agentscited by 4
- 2025Learning Probabilistic Temporal Logic Specifications for Stochastic Systemscited by 2
- 2023Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learningcited by 1
- 2025Know your Trajectory -- Trustworthy Reinforcement Learning deployment through Importance-Based Trajectory Analysiscited by 0
- 2026A Formal Framework for the Explanation of Finite Automata Decisionscited by 0
- 2026ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policiescited by 0
- 2026ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies8 Jul 2026
- 2026A Formal Framework for the Explanation of Finite Automata Decisions12 Feb 2026
- 2025Know your Trajectory -- Trustworthy Reinforcement Learning deployment through Importance-Based Trajectory Analysis7 Dec 2025
- 2025Learning Probabilistic Temporal Logic Specifications for Stochastic Systems17 May 2025
- 2023Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning11 Jul 2023
- 2023Explainable Multi-Agent Reinforcement Learning for Temporal Queries17 May 2023
- 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges17 May 2023
- 2023Explaining RL Decisions with Trajectories6 May 2023
- 2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanations24 Feb 2023
- 2022A Survey of Explainable Reinforcement Learning17 Feb 2022
- 2022Local Explanations for Reinforcement Learning8 Feb 2022
- 2021Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents18 Jan 2021