# Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learning

arXiv: https://arxiv.org/abs/2601.15086 · HTML: https://mlsanity.com/paper/2601.15086/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Partially Observable Markov Decision Processes in Robotics: A Survey](https://mlsanity.com/paper/2209.10342/) (2022)
- [DeepMind Lab](https://mlsanity.com/paper/1612.03801/) (2016)
- [When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment](https://mlsanity.com/paper/2307.03864/) (2023)
- [POPGym: Benchmarking Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2303.01859/) (2023)
- [SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation](https://mlsanity.com/paper/2501.18564/) (2025)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [Partially Observable Markov Decision Processes (POMDPs) and Robotics](https://mlsanity.com/paper/2107.07599/) (2021)
- [Evaluating Long-Term Memory in 3D Mazes](https://mlsanity.com/paper/2210.13383/) (2022)
- [Towards mental time travel: a hierarchical memory for reinforcement learning agents](https://mlsanity.com/paper/2105.14039/) (2021)
- [Deep Transformer Q-Networks for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2206.01078/) (2022)
- [Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning](https://mlsanity.com/paper/2502.10550/) (2025)
- [Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents](https://mlsanity.com/paper/2309.17207/) (2023)
- [Explain My Surprise: Learning Efficient Long-Term Memory by Predicting Uncertain Outcomes](https://mlsanity.com/paper/2207.13649/) (2022)
- [Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization](https://mlsanity.com/paper/2205.11104/) (2022)
- [Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning](https://mlsanity.com/paper/2410.10132/) (2024)
- [Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains](https://mlsanity.com/paper/2508.00046/) (2025)
- [Control Theory Meets POMDPs: A Hybrid Systems Approach](https://mlsanity.com/paper/1905.08095/) (2019)
- [Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation](https://mlsanity.com/paper/2412.06531/) (2024)

## Built on by (most cited first, 1 of 1)

- [ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learning](https://mlsanity.com/paper/2610.00592/) (2026)

## Abstract (arXiv)

Effective decision-making in the real world depends on memory that is both stable and adaptive: environments change over time, and agents must retain relevant information over long horizons while also updating or overwriting outdated content when circumstances shift. Existing Reinforcement Learning (RL) benchmarks and memory-augmented agents focus primarily on retention, leaving the equally critical ability of memory rewriting largely unexplored. To address this gap, we introduce a benchmark that explicitly tests continual memory updating under partial observability, i.e. the natural setting where an agent must rely on memory rather than current observations, and use it to compare recurrent, transformer-based, and structured memory architectures. Our experiments reveal that classic recurrent models, despite their simplicity, demonstrate greater flexibility and robustness in memory rewriting tasks than modern structured memories, which succeed only under narrow conditions, and transformer-based agents, which often fail beyond trivial retention cases. These findings expose a fundamental limitation of current approaches and emphasize the necessity of memory mechanisms that balance stable retention with adaptive updating. Our work highlights this overlooked challenge, introduces benchmarks to evaluate it, and offers insights for designing future RL agents with explicit and trainable forgetting mechanisms. Code: https://quartz-admirer.github.io/Memory-Rewriting/