mlsanityData to 7 Oct 2026

Generalizable Episodic Memory for Deep Reinforcement Learning

arXiv 2103.06469 ↗First posted 11 Mar 2021cs.LG cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Episodic memory-based methods can rapidly latch onto past successful strategies by a non-parametric memory and improve sample efficiency of traditional reinforcement learning. However, little effort is put into the continuous domain, where a state is never visited twice, and previous episodic methods fail to efficiently aggregate experience across trajectories. To address this problem, we propose Generalizable Episodic Memory (GEM), which effectively organizes the state-action values of episodic memory in a generalizable manner and supports implicit planning on memorized trajectories. GEM utilizes a double estimator to reduce the overestimation bias induced by value propagation in the planning process. Empirical evaluation shows that our method significantly outperforms existing trajectory-based methods on various MuJoCo continuous control tasks. To further show the general applicability, we evaluate our method on Atari games with discrete action space, which also shows a significant improvement over baseline algorithms.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 23 of its 49 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

23 of 49 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2016OpenAI Gymcited by 1,684
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  12. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
11 more
  1. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  2. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  3. 2017Neural Episodic Controlcited by 148
  4. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  5. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  6. 2016Model-Free Episodic Controlcited by 100
  7. 2018Episodic Memory Deep Q-Networkscited by 41
  8. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
  9. 2016Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tighteningcited by 21
  10. 2018Fast deep reinforcement learning using online adjustments from the pastcited by 14
  11. 2017Convergent Tree Backup and Retrace with Function Approximationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 6; 2024: 2; 2025: 1; 2026: 3.

22 papers.

Sort:

By area:

  1. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  2. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  3. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  4. 2021Model-Based Episodic Memory Induces Dynamic Hybrid Controlscited by 14
  5. 2022Beyond Supervised Continual Learning: a Reviewcited by 11
  6. 2022BATS: Best Action Trajectory Stitchingcited by 9
  7. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
  8. 2023Neural Episodic Control with State Abstractioncited by 7
  9. 2021On the Estimation Bias in Double Q-Learningcited by 4
  10. 2024Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learningcited by 4
  11. 2021Successor Feature Neural Episodic Controlcited by 3
  12. 2023EMO: Episodic Memory Optimization for Few-Shot Meta-Learningcited by 2
10 more
  1. 2025RANa: Retrieval-Augmented Navigationcited by 2
  2. 2022Off-Beat Multi-Agent Reinforcement Learningcited by 1
  3. 2022Continuous Episodic Controlcited by 1
  4. 2023Two-Memory Reinforcement Learningcited by 1
  5. 2023Deep Reinforcement Learning with Task-Adaptive Retrieval via Hypernetworkcited by 0
  6. 2023Policy Optimization with Smooth Guidance Learned from State-Only Demonstrationscited by 0
  7. 2024Highway Graph to Accelerate Reinforcement Learningcited by 0
  8. 2026Artifacts as Memory Beyond the Agent Boundarycited by 0
  9. 2026Repetition as Reinforcement: Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learningcited by 0
  10. 2026CMDO: A Cognitive Memory-Driven Optimization Algorithm for Adaptive Population-Based Searchcited by 0