# Deep Reinforcement Learning Discovers Internal Models

arXiv: https://arxiv.org/abs/1606.05174 · HTML: https://mlsanity.com/paper/1606.05174/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2016.

## Built on by (most cited first, 3 of 3)

- [DRL: Deep Reinforcement Learning for Intelligent Robot Control -- Concept, Literature, and Future](https://mlsanity.com/paper/2105.13806/) (2021)
- [Utilizing Explainability Techniques for Reinforcement Learning Model Assurance](https://mlsanity.com/paper/2311.15838/) (2023)
- [A Framework for Adversarial Analysis of Decision Support Systems Prior to Deployment](https://mlsanity.com/paper/2505.21414/) (2025)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) is a trending field of research, showing great promise in challenging problems such as playing Atari, solving Go and controlling robots. While DRL agents perform well in practice we are still lacking the tools to analayze their performance. In this work we present the Semi-Aggregated MDP (SAMDP) model. A model best suited to describe policies exhibiting both spatial and temporal hierarchies. We describe its advantages for analyzing trained policies over other modeling approaches, and show that under the right state representation, like that of DQN agents, SAMDP can help to identify skills. We detail the automatic process of creating it from recorded trajectories, up to presenting it on t-SNE maps. We explain how to evaluate its fitness and show surprising results indicating high compatibility with the policy at hand. We conclude by showing how using the SAMDP model, an extra performance gain can be squeezed from the agent.