# Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning

arXiv: https://arxiv.org/abs/2305.02054 · HTML: https://mlsanity.com/paper/2305.02054/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 3 May 2023.

## Builds on (most cited first, 22 of 22)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Remember and Forget for Experience Replay](https://mlsanity.com/paper/1807.05827/) (2018)
- [Superhuman Artificial Intelligence Can Improve Human Decision Making by Increasing Novelty](https://mlsanity.com/paper/2303.07462/) (2023)
- [Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2](https://mlsanity.com/paper/2208.05056/) (2022)
- [Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning](https://mlsanity.com/paper/1810.11388/) (2018)
- [Behavior Self-Organization Supports Task Inference for Continual Robot Learning](https://mlsanity.com/paper/2107.04533/) (2021)
- [Efficient Intrinsically Motivated Robotic Grasping with Learning-Adaptive Imagination in Latent Space](https://mlsanity.com/paper/1910.04729/) (2019)
- [Stratified Experience Replay: Correcting Multiplicity Bias in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2102.11319/) (2021)
- [Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorations](https://mlsanity.com/paper/2208.02680/) (2022)
- [Bootstrapping a DQN Replay Memory with Synthetic Experiences](https://mlsanity.com/paper/2002.01370/) (2020)
- [Learning to Autonomously Reach Objects with NICO and Grow-When-Required Networks](https://mlsanity.com/paper/2210.07851/) (2022)

## Built on by (most cited first, 6 of 6)

- [A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances](https://mlsanity.com/paper/2505.17799/) (2025)
- [Towards Incremental Learning in Large Language Models: A Critical Review](https://mlsanity.com/paper/2404.18311/) (2024)
- [Class Incremental Learning with Task-Specific Batch Normalization and Out-of-Distribution Detection](https://mlsanity.com/paper/2411.00430/) (2024)
- [Reinforcement Learning Increases Wind Farm Power Production by Enabling Closed-Loop Collaborative Control](https://mlsanity.com/paper/2506.20554/) (2025)
- [Natural Mitigation of Catastrophic Interference: Continual Learning in Power-Law Learning Environments](https://mlsanity.com/paper/2401.10393/) (2024)
- [An Adaptive Resonance Theory-based Topological Clustering Algorithm with a Self-Adjusting Vigilance Parameter](https://mlsanity.com/paper/2511.17983/) (2025)

## Abstract (arXiv)

Deep Reinforcement Learning agents often suffer from catastrophic forgetting, forgetting previously found solutions in parts of the input space when training on new data. Replay Memories are a common solution to the problem, decorrelating and shuffling old and new training samples. They naively store state transitions as they come in, without regard for redundancy. We introduce a novel cognitive-inspired replay memory approach based on the Grow-When-Required (GWR) self-organizing network, which resembles a map-based mental model of the world. Our approach organizes stored transitions into a concise environment-model-like network of state-nodes and transition-edges, merging similar samples to reduce the memory size and increase pair-wise distance among samples, which increases the relevancy of each sample. Overall, our paper shows that map-based experience replay allows for significant memory reduction with only small performance decreases.