# Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learning

arXiv: https://arxiv.org/abs/2402.09290 · HTML: https://mlsanity.com/paper/2402.09290/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Provably Efficient Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/1907.05388/) (2019)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms](https://mlsanity.com/paper/2102.00815/) (2021)
- [Generalization and Regularization in DQN](https://mlsanity.com/paper/1810.00123/) (2018)
- [Offline Reinforcement Learning from Images with Latent Space Models](https://mlsanity.com/paper/2012.11547/) (2020)
- [Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction](https://mlsanity.com/paper/2007.14535/) (2020)
- [Combining Optimal Control and Learning for Visual Navigation in Novel Environments](https://mlsanity.com/paper/1903.02531/) (2019)
- [Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships](https://mlsanity.com/paper/2011.04251/) (2020)
- [Deep Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1811.11329/) (2018)
- [Unbiased Asymmetric Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2105.11674/) (2021)
- [Learning the Linear Quadratic Regulator from Nonlinear Observations](https://mlsanity.com/paper/2010.03799/) (2020)
- [Robust Guarantees for Perception-Based Control](https://mlsanity.com/paper/1907.03680/) (2019)
- [Learning in POMDPs is Sample-Efficient with Hindsight Observability](https://mlsanity.com/paper/2301.13857/) (2023)
- [Simultaneous Policy Learning and Latent State Inference for Imitating Driver Behavior](https://mlsanity.com/paper/1704.05566/) (2017)
- [Inherently Explainable Reinforcement Learning in Natural Language](https://mlsanity.com/paper/2112.08907/) (2021)
- [Explainable Reinforcement Learning via Model Transforms](https://mlsanity.com/paper/2209.12006/) (2022)
- [Bootstrapped Representations in Reinforcement Learning](https://mlsanity.com/paper/2306.10171/) (2023)
- [Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2205.14557/) (2022)
- [Provably Efficient Representation Learning with Tractable Planning in Low-Rank POMDP](https://mlsanity.com/paper/2306.12356/) (2023)
- [Local Explanations for Reinforcement Learning](https://mlsanity.com/paper/2202.03597/) (2022)

## Built on by (most cited first, 2 of 2)

- [Perception-Based Beliefs for POMDPs with Visual Observations](https://mlsanity.com/paper/2602.05679/) (2026)
- [Learning Causal States Under Partial Observability and Perturbation](https://mlsanity.com/paper/2512.00357/) (2025)

## Abstract (arXiv)

Deep reinforcement learning has demonstrated remarkable achievements across diverse domains such as video games, robotic control, autonomous driving, and drug discovery. Common methodologies in partially-observable domains largely lean on end-to-end learning from high-dimensional observations, such as images, without explicitly reasoning about true state. We suggest an alternative direction, introducing the Partially Supervised Reinforcement Learning (PSRL) framework. At the heart of PSRL is the fusion of both supervised and unsupervised learning. The approach leverages a state estimator to distill supervised semantic state information from high-dimensional observations which are often fully observable at training time. This yields more interpretable policies that compose state predictions with control. In parallel, it captures an unsupervised latent representation. These two-the semantic state and the latent state-are then fused and utilized as inputs to a policy network. This juxtaposition offers practitioners a flexible and dynamic spectrum: from emphasizing supervised state information to integrating richer, latent insights. Extensive experimental results indicate that by merging these dual representations, PSRL offers a potent balance, enhancing model interpretability while preserving, and often significantly outperforming, the performance benchmarks set by traditional methods in terms of reward and convergence speed.