# On Improving Deep Reinforcement Learning for POMDPs

arXiv: https://arxiv.org/abs/1804.06309 · HTML: https://mlsanity.com/paper/1804.06309/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2018.

## Built on by (most cited first, 30 of 50)

- [Federated Reinforcement Learning: Techniques, Applications, and Open Challenges](https://mlsanity.com/paper/2108.11887/) (2021)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [Provable Reinforcement Learning with a Short-Term Memory](https://mlsanity.com/paper/2202.03983/) (2022)
- [Deep Transformer Q-Networks for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2206.01078/) (2022)
- [Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning](https://mlsanity.com/paper/2502.10550/) (2025)
- [Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition](https://mlsanity.com/paper/2105.08692/) (2021)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [Structured World Belief for Reinforcement Learning in POMDP](https://mlsanity.com/paper/2107.08577/) (2021)
- [Recurrent Off-policy Baselines for Memory-based Continuous Control](https://mlsanity.com/paper/2110.12628/) (2021)
- [Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains](https://mlsanity.com/paper/2110.15949/) (2021)
- [A Bayesian Framework for Digital Twin-Based Control, Monitoring, and Data Collection in Wireless Systems](https://mlsanity.com/paper/2212.01351/) (2022)
- [POMDP inference and robust solution via deep reinforcement learning: An application to railway optimal maintenance](https://mlsanity.com/paper/2307.08082/) (2023)
- [Recurrent networks, hidden states and beliefs in partially observable environments](https://mlsanity.com/paper/2208.03520/) (2022)
- [Bridging POMDPs and Bayesian decision making for robust maintenance planning under model uncertainty: An application to railway systems](https://mlsanity.com/paper/2212.07933/) (2022)
- [Informed POMDP: Leveraging Additional Information in Model-Based RL](https://mlsanity.com/paper/2306.11488/) (2023)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Provable Representation with Efficient Planning for Partial Observable Reinforcement Learning](https://mlsanity.com/paper/2311.12244/) (2023)
- [Reinforcement Learning in Presence of Discrete Markovian Context Evolution](https://mlsanity.com/paper/2202.06557/) (2022)
- [Discovering Exfiltration Paths Using Reinforcement Learning with Attack Graphs](https://mlsanity.com/paper/2201.12416/) (2022)
- [SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2105.06228/) (2021)
- [Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions](https://mlsanity.com/paper/2409.13144/) (2024)
- [A Theoretical Justification for Asymmetric Actor-Critic Algorithms](https://mlsanity.com/paper/2501.19116/) (2025)
- [Generalized Beliefs for Cooperative AI](https://mlsanity.com/paper/2206.12765/) (2022)
- [Cochain Perspectives on Temporal-Difference Signals for Learning Beyond Markov Dynamics](https://mlsanity.com/paper/2602.06939/) (2026)
- [How Are Learned Perception-Based Controllers Impacted by the Limits of Robust Control?](https://mlsanity.com/paper/2104.00827/) (2021)
- [Scientific Discovery and the Cost of Measurement -- Balancing Information and Cost in Reinforcement Learning](https://mlsanity.com/paper/2112.07535/) (2021)
- [On the Challenges of using Reinforcement Learning in Precision Drug Dosing: Delay and Prolongedness of Action Effects](https://mlsanity.com/paper/2301.00512/) (2023)
- [Towards Cost Sensitive Decision Making](https://mlsanity.com/paper/2410.03892/) (2024)
- [Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access](https://mlsanity.com/paper/2509.26000/) (2025)
- [Uncertainty Maximization in Partially Observable Domains: A Cognitive Perspective](https://mlsanity.com/paper/2102.11232/) (2021)

## Abstract (arXiv)

Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little work has been done in deep RL to handle partially observable environments. We propose a new architecture called Action-specific Deep Recurrent Q-Network (ADRQN) to enhance learning performance in partially observable domains. Actions are encoded by a fully connected layer and coupled with a convolutional observation to form an action-observation pair. The time series of action-observation pairs are then integrated by an LSTM layer that learns latent states based on which a fully connected layer computes Q-values as in conventional Deep Q-Networks (DQNs). We demonstrate the effectiveness of our new architecture in several partially observable domains, including flickering Atari games.