# Generalizable Episodic Memory for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2103.06469 · HTML: https://mlsanity.com/paper/2103.06469/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 11 Mar 2021.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics](https://mlsanity.com/paper/2005.04269/) (2020)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Maxmin Q-learning: Controlling the Estimation Bias of Q-learning](https://mlsanity.com/paper/2002.06487/) (2020)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [Model-Free Episodic Control](https://mlsanity.com/paper/1606.04460/) (2016)
- [Episodic Memory Deep Q-Networks](https://mlsanity.com/paper/1805.07603/) (2018)
- [Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update](https://mlsanity.com/paper/1805.12375/) (2018)
- [Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening](https://mlsanity.com/paper/1611.01606/) (2016)
- [Fast deep reinforcement learning using online adjustments from the past](https://mlsanity.com/paper/1810.08163/) (2018)
- [Convergent Tree Backup and Retrace with Function Approximation](https://mlsanity.com/paper/1705.09322/) (2017)

## Built on by (most cited first, 23 of 23)

- [Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling](https://mlsanity.com/paper/2209.14548/) (2022)
- [Offline Reinforcement Learning with Value-based Episodic Memory](https://mlsanity.com/paper/2110.09796/) (2021)
- [Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy](https://mlsanity.com/paper/2207.12141/) (2022)
- [Model-Based Episodic Memory Induces Dynamic Hybrid Controls](https://mlsanity.com/paper/2111.02104/) (2021)
- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [BATS: Best Action Trajectory Stitching](https://mlsanity.com/paper/2204.12026/) (2022)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Neural Episodic Control with State Abstraction](https://mlsanity.com/paper/2301.11490/) (2023)
- [Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.01112/) (2024)
- [On the Estimation Bias in Double Q-Learning](https://mlsanity.com/paper/2109.14419/) (2021)
- [Successor Feature Neural Episodic Control](https://mlsanity.com/paper/2111.03110/) (2021)
- [EMO: Episodic Memory Optimization for Few-Shot Meta-Learning](https://mlsanity.com/paper/2306.05189/) (2023)
- [RANa: Retrieval-Augmented Navigation](https://mlsanity.com/paper/2504.03524/) (2025)
- [Off-Beat Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.13718/) (2022)
- [Continuous Episodic Control](https://mlsanity.com/paper/2211.15183/) (2022)
- [Two-Memory Reinforcement Learning](https://mlsanity.com/paper/2304.10098/) (2023)
- [Deep Reinforcement Learning with Task-Adaptive Retrieval via Hypernetwork](https://mlsanity.com/paper/2306.10698/) (2023)
- [Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations](https://mlsanity.com/paper/2401.00162/) (2023)
- [Highway Graph to Accelerate Reinforcement Learning](https://mlsanity.com/paper/2405.11727/) (2024)
- [Learning From Failures: Efficient Reinforcement Learning Control with Episodic Memory](https://mlsanity.com/paper/2603.07110/) (2026)
- [Artifacts as Memory Beyond the Agent Boundary](https://mlsanity.com/paper/2604.08756/) (2026)
- [Repetition as Reinforcement: Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learning](https://mlsanity.com/paper/2608.17347/) (2026)
- [CMDO: A Cognitive Memory-Driven Optimization Algorithm for Adaptive Population-Based Search](https://mlsanity.com/paper/2609.35657/) (2026)

## Abstract (arXiv)

Episodic memory-based methods can rapidly latch onto past successful strategies by a non-parametric memory and improve sample efficiency of traditional reinforcement learning. However, little effort is put into the continuous domain, where a state is never visited twice, and previous episodic methods fail to efficiently aggregate experience across trajectories. To address this problem, we propose Generalizable Episodic Memory (GEM), which effectively organizes the state-action values of episodic memory in a generalizable manner and supports implicit planning on memorized trajectories. GEM utilizes a double estimator to reduce the overestimation bias induced by value propagation in the planning process. Empirical evaluation shows that our method significantly outperforms existing trajectory-based methods on various MuJoCo continuous control tasks. To further show the general applicability, we evaluate our method on Atari games with discrete action space, which also shows a significant improvement over baseline algorithms.