# Learning What to Memorize: Using Intrinsic Motivation to Form Useful Memory in Partially Observable Reinforcement Learning

arXiv: https://arxiv.org/abs/2110.12810 · HTML: https://mlsanity.com/paper/2110.12810/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [A survey on intrinsic motivation in reinforcement learning](https://mlsanity.com/paper/1908.06976/) (2019)
- [On Improving Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1804.06309/) (2018)
- [Solving POMDPs by Searching the Space of Finite Policies](https://mlsanity.com/paper/1301.6720/) (2013)
- [Learning Finite-State Controllers for Partially Observable Environments](https://mlsanity.com/paper/1301.6721/) (2013)
- [Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration](https://mlsanity.com/paper/1903.07400/) (2019)
- [The act of remembering: a study in partially observable reinforcement learning](https://mlsanity.com/paper/2010.01753/) (2020)
- [Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes](https://mlsanity.com/paper/1805.04419/) (2018)
- [Reinforcement Learning in POMDPs with Memoryless Options and Option-Observation Initiation Sets](https://mlsanity.com/paper/1708.06551/) (2017)

## Built on by (most cited first, 1 of 1)

- [Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments](https://mlsanity.com/paper/2512.19154/) (2025)

## Abstract (arXiv)

Reinforcement Learning faces an important challenge in partial observable environments that has long-term dependencies. In order to learn in an ambiguous environment, an agent has to keep previous perceptions in a memory. Earlier memory based approaches use a fixed method to determine what to keep in the memory, which limits them to certain problems. In this study, we follow the idea of giving the control of the memory to the agent by allowing it to have memory-changing actions. This learning mechanism is supported by an intrinsic motivation to memorize rare observations that can help the agent to disambiguate its state in the environment. Our approach is experimented and analyzed on several partial observable tasks with long-term dependencies and compared with other memory based methods.