# Memory-based control with recurrent neural networks

arXiv: https://arxiv.org/abs/1512.04455 · HTML: https://mlsanity.com/paper/1512.04455/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2015.

## Built on by (most cited first, 30 of 78)

- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Federated Reinforcement Learning: Techniques, Applications, and Open Challenges](https://mlsanity.com/paper/2108.11887/) (2021)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents](https://mlsanity.com/paper/2310.09971/) (2023)
- [Unsupervised Visual Attention and Invariance for Reinforcement Learning](https://mlsanity.com/paper/2104.02921/) (2021)
- [The Free Energy Principle for Perception and Action: A Deep Learning Perspective](https://mlsanity.com/paper/2207.06415/) (2022)
- [Recurrent Off-policy Baselines for Memory-based Continuous Control](https://mlsanity.com/paper/2110.12628/) (2021)
- [Equivariant Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2408.14336/) (2024)
- [Recurrent Neural Network Controllers Synthesis with Stability Guarantees for Partially Observed Systems](https://mlsanity.com/paper/2109.03861/) (2021)
- [Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents](https://mlsanity.com/paper/2309.17207/) (2023)
- [AdaptNet: Policy Adaptation for Physics-Based Character Control](https://mlsanity.com/paper/2310.00239/) (2023)
- [Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximation](https://mlsanity.com/paper/2106.07472/) (2021)
- [Recurrent networks, hidden states and beliefs in partially observable environments](https://mlsanity.com/paper/2208.03520/) (2022)
- [Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challenges](https://mlsanity.com/paper/2205.14495/) (2022)
- [Informed POMDP: Leveraging Additional Information in Model-Based RL](https://mlsanity.com/paper/2306.11488/) (2023)
- [The Evolution of Reinforcement Learning in Quantitative Finance: A Survey](https://mlsanity.com/paper/2408.10932/) (2024)
- [Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization](https://mlsanity.com/paper/2205.11104/) (2022)
- [Provable Representation with Efficient Planning for Partial Observable Reinforcement Learning](https://mlsanity.com/paper/2311.12244/) (2023)
- [ODE-based Recurrent Model-free Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2309.14078/) (2023)
- [Offline RL with Observation Histories: Analyzing and Improving Sample Complexity](https://mlsanity.com/paper/2310.20663/) (2023)
- [Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy](https://mlsanity.com/paper/2407.07333/) (2024)
- [Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review](https://mlsanity.com/paper/2208.12328/) (2022)
- [A Theoretical Justification for Asymmetric Actor-Critic Algorithms](https://mlsanity.com/paper/2501.19116/) (2025)
- [Zero-Shot Reinforcement Learning Under Partial Observability](https://mlsanity.com/paper/2506.15446/) (2025)
- [Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2409.16824/) (2024)
- [Self-optimizing adaptive optics control with Reinforcement Learning for high-contrast imaging](https://mlsanity.com/paper/2108.11332/) (2021)
- [RL-PGO: Reinforcement Learning-based Planar Pose-Graph Optimization](https://mlsanity.com/paper/2202.13221/) (2022)
- [The configurable tree graph (CT-graph): measurable problems in partially observable and distal reward environments for lifelong reinforcement learning](https://mlsanity.com/paper/2302.10887/) (2023)
- [PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Tasks](https://mlsanity.com/paper/2307.05891/) (2023)

## Abstract (arXiv)

Partially observed control problems are a challenging aspect of reinforcement learning. We extend two related, model-free algorithms for continuous control -- deterministic policy gradient and stochastic value gradient -- to solve partially observed domains using recurrent neural networks trained with backpropagation through time. We demonstrate that this approach, coupled with long-short term memory is able to solve a variety of physical control problems exhibiting an assortment of memory requirements. These include the short-term integration of information from noisy sensors and the identification of system parameters, as well as long-term memory problems that require preserving information over many time steps. We also demonstrate success on a combined exploration and memory problem in the form of a simplified version of the well-known Morris water maze task. Finally, we show that our approach can deal with high-dimensional observations by learning directly from pixels. We find that recurrent deterministic and stochastic policies are able to learn similarly good solutions to these tasks, including the water maze where the agent must learn effective search strategies.