# Learning Deep Neural Network Policies with Continuous Memory States

arXiv: https://arxiv.org/abs/1507.01273 · HTML: https://mlsanity.com/paper/1507.01273/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2015.

## Built on by (most cited first, 4 of 4)

- [Offline RL with Observation Histories: Analyzing and Improving Sample Complexity](https://mlsanity.com/paper/2310.20663/) (2023)
- [When Can Model-Free Reinforcement Learning be Enough for Thinking?](https://mlsanity.com/paper/2506.17124/) (2025)
- [Generalized Beliefs for Cooperative AI](https://mlsanity.com/paper/2206.12765/) (2022)
- [MEMBOT: Memory-Based Robot in Intermittent POMDP](https://mlsanity.com/paper/2509.11225/) (2025)

## Abstract (arXiv)

Policy learning for partially observed control tasks requires policies that can remember salient information from past observations. In this paper, we present a method for learning policies with internal memory for high-dimensional, continuous systems, such as robotic manipulators. Our approach consists of augmenting the state and action space of the system with continuous-valued memory states that the policy can read from and write to. Learning general-purpose policies with this type of memory representation directly is difficult, because the policy must automatically figure out the most salient information to memorize at each time step. We show that, by decomposing this policy search problem into a trajectory optimization phase and a supervised learning phase through a method called guided policy search, we can acquire policies with effective memorization and recall strategies. Intuitively, the trajectory optimization phase chooses the values of the memory states that will make it easier for the policy to produce the right action in future states, while the supervised learning phase encourages the policy to use memorization actions to produce those memory states. We evaluate our method on tasks involving continuous control in manipulation and navigation settings, and show that our method can learn complex policies that successfully complete a range of tasks that require memory.