# Episodic Memory Deep Q-Networks

arXiv: https://arxiv.org/abs/1805.07603 · HTML: https://mlsanity.com/paper/1805.07603/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2018.

## Built on by (most cited first, 30 of 42)

- [Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration](https://mlsanity.com/paper/2111.11032/) (2021)
- [Offline Reinforcement Learning with Value-based Episodic Memory](https://mlsanity.com/paper/2110.09796/) (2021)
- [Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning](https://mlsanity.com/paper/2502.10550/) (2025)
- [Large-Scale Retrieval for Reinforcement Learning](https://mlsanity.com/paper/2206.05314/) (2022)
- [Generalizable Episodic Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/2103.06469/) (2021)
- [AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition](https://mlsanity.com/paper/2512.03794/) (2025)
- [DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs](https://mlsanity.com/paper/2010.08891/) (2020)
- [Model-Based Episodic Memory Induces Dynamic Hybrid Controls](https://mlsanity.com/paper/2111.02104/) (2021)
- [Theoretically Principled Deep RL Acceleration via Nearest Neighbor Function Approximation](https://mlsanity.com/paper/2110.04422/) (2021)
- [UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience](https://mlsanity.com/paper/2603.24533/) (2026)
- [A Machine with Short-Term, Episodic, and Semantic Memory Systems](https://mlsanity.com/paper/2212.02098/) (2022)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Solving Continuous Control with Episodic Memory](https://mlsanity.com/paper/2106.08832/) (2021)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [Neural Episodic Control with State Abstraction](https://mlsanity.com/paper/2301.11490/) (2023)
- [MacroHFT: Memory Augmented Context-aware Reinforcement Learning On High Frequency Trading](https://mlsanity.com/paper/2406.14537/) (2024)
- [Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.01112/) (2024)
- [MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs](https://mlsanity.com/paper/2605.08374/) (2026)
- [On the Estimation Bias in Double Q-Learning](https://mlsanity.com/paper/2109.14419/) (2021)
- [Memory poisoning and secure multi-agent systems](https://mlsanity.com/paper/2603.20357/) (2026)
- [Successor Feature Neural Episodic Control](https://mlsanity.com/paper/2111.03110/) (2021)
- [Sequential memory improves sample and memory efficiency in Episodic Control](https://mlsanity.com/paper/2112.14734/) (2021)
- [Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization](https://mlsanity.com/paper/2302.02299/) (2023)
- [Episodic Reinforcement Learning with Expanded State-reward Space](https://mlsanity.com/paper/2401.10516/) (2024)
- [Learning Diverse Policies with Soft Self-Generated Guidance](https://mlsanity.com/paper/2402.04539/) (2024)
- [Enhancing Q-Learning with Large Language Model Heuristics](https://mlsanity.com/paper/2405.03341/) (2024)
- [RANa: Retrieval-Augmented Navigation](https://mlsanity.com/paper/2504.03524/) (2025)
- [State-based Episodic Memory for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2110.09817/) (2021)
- [Continuous Episodic Control](https://mlsanity.com/paper/2211.15183/) (2022)
- [Two-Memory Reinforcement Learning](https://mlsanity.com/paper/2304.10098/) (2023)

## Abstract (arXiv)

Reinforcement learning (RL) algorithms have made huge progress in recent years by leveraging the power of deep neural networks (DNN). Despite the success, deep RL algorithms are known to be sample inefficient, often requiring many rounds of interaction with the environments to obtain satisfactory performance. Recently, episodic memory based RL has attracted attention due to its ability to latch on good actions quickly. In this paper, we present a simple yet effective biologically inspired RL algorithm called Episodic Memory Deep Q-Networks (EMDQN), which leverages episodic memory to supervise an agent during training. Experiments show that our proposed method can lead to better sample efficiency and is more likely to find good policies. It only requires 1/5 of the interactions of DQN to achieve many state-of-the-art performances on Atari games, significantly outperforming regular DQN and other episodic memory based RL algorithms.