# Solving Continuous Control with Episodic Memory

arXiv: https://arxiv.org/abs/2106.08832 · HTML: https://mlsanity.com/paper/2106.08832/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Model-Free Episodic Control](https://mlsanity.com/paper/1606.04460/) (2016)
- [Episodic Memory Deep Q-Networks](https://mlsanity.com/paper/1805.07603/) (2018)
- [Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments](https://mlsanity.com/paper/1903.00827/) (2019)

## Built on by (most cited first, 8 of 8)

- [Neural Episodic Control with State Abstraction](https://mlsanity.com/paper/2301.11490/) (2023)
- [Episodic Reinforcement Learning with Expanded State-reward Space](https://mlsanity.com/paper/2401.10516/) (2024)
- [Dejavu: Towards Experience Feedback Learning for Embodied Intelligence](https://mlsanity.com/paper/2510.10181/) (2025)
- [Guided Exploration in Reinforcement Learning via Monte Carlo Critic Optimization](https://mlsanity.com/paper/2206.12674/) (2022)
- [Continuous Episodic Control](https://mlsanity.com/paper/2211.15183/) (2022)
- [Two-Memory Reinforcement Learning](https://mlsanity.com/paper/2304.10098/) (2023)
- [Deep Reinforcement Learning with Task-Adaptive Retrieval via Hypernetwork](https://mlsanity.com/paper/2306.10698/) (2023)
- [Learning From Failures: Efficient Reinforcement Learning Control with Episodic Memory](https://mlsanity.com/paper/2603.07110/) (2026)

## Abstract (arXiv)

Episodic memory lets reinforcement learning algorithms remember and exploit promising experience from the past to improve agent performance. Previous works on memory mechanisms show benefits of using episodic-based data structures for discrete action problems in terms of sample-efficiency. The application of episodic memory for continuous control with a large action space is not trivial. Our study aims to answer the question: can episodic memory be used to improve agent's performance in continuous control? Our proposed algorithm combines episodic memory with Actor-Critic architecture by modifying critic's objective. We further improve performance by introducing episodic-based replay buffer prioritization. We evaluate our algorithm on OpenAI gym domains and show greater sample-efficiency compared with the state-of-the art model-free off-policy algorithms.