# Model-Free Episodic Control

arXiv: https://arxiv.org/abs/1606.04460 · HTML: https://mlsanity.com/paper/1606.04460/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2016.

## Built on by (most cited first, 30 of 102)

- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling](https://mlsanity.com/paper/2209.14548/) (2022)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [Human-inspired Episodic Memory for Infinite Context LLMs](https://mlsanity.com/paper/2407.09450/) (2024)
- [Retrieval-Augmented Reinforcement Learning](https://mlsanity.com/paper/2202.08417/) (2022)
- [Larimar: Large Language Models with Episodic Memory Control](https://mlsanity.com/paper/2403.11901/) (2024)
- [Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration](https://mlsanity.com/paper/2111.11032/) (2021)
- [Offline Reinforcement Learning with Value-based Episodic Memory](https://mlsanity.com/paper/2110.09796/) (2021)
- [Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods](https://mlsanity.com/paper/2102.13045/) (2021)
- [Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments](https://mlsanity.com/paper/2101.08152/) (2021)
- [Generative Adversarial Self-Imitation Learning](https://mlsanity.com/paper/1812.00950/) (2018)
- [Large-Scale Retrieval for Reinforcement Learning](https://mlsanity.com/paper/2206.05314/) (2022)
- [Generalizable Episodic Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/2103.06469/) (2021)
- [Retrieval-Augmented Decision Transformer: External Memory for In-context RL](https://mlsanity.com/paper/2410.07071/) (2024)
- [Natural Language Reinforcement Learning](https://mlsanity.com/paper/2402.07157/) (2024)
- [DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs](https://mlsanity.com/paper/2010.08891/) (2020)
- [On the link between conscious function and general intelligence in humans and machines](https://mlsanity.com/paper/2204.05133/) (2022)
- [Episodic Novelty Through Temporal Distance](https://mlsanity.com/paper/2501.15418/) (2025)
- [ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-Learning](https://mlsanity.com/paper/2309.13701/) (2023)
- [Model-Based Episodic Memory Induces Dynamic Hybrid Controls](https://mlsanity.com/paper/2111.02104/) (2021)
- [R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory](https://mlsanity.com/paper/2501.12485/) (2025)
- [Self-Imitation Learning via Generalized Lower Bound Q-learning](https://mlsanity.com/paper/2006.07442/) (2020)
- [Self-Evolving World Models for LLM Agent Planning](https://mlsanity.com/paper/2606.30639/) (2026)
- [Neuro-algorithmic Policies enable Fast Combinatorial Generalization](https://mlsanity.com/paper/2102.07456/) (2021)
- [Theoretically Principled Deep RL Acceleration via Nearest Neighbor Function Approximation](https://mlsanity.com/paper/2110.04422/) (2021)
- [Episodic memory in AI agents poses risks that should be studied and mitigated](https://mlsanity.com/paper/2501.11739/) (2025)
- [Evolution and The Knightian Blindspot of Machine Learning](https://mlsanity.com/paper/2501.13075/) (2025)
- [BATS: Best Action Trajectory Stitching](https://mlsanity.com/paper/2204.12026/) (2022)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)

## Abstract (arXiv)

State of the art deep reinforcement learning algorithms take many millions of interactions to attain human-level performance. Humans, on the other hand, can very quickly exploit highly rewarding nuances of an environment upon first discovery. In the brain, such rapid learning is thought to depend on the hippocampus and its capacity for episodic memory. Here we investigate whether a simple model of hippocampal episodic control can learn to solve difficult sequential decision-making tasks. We demonstrate that it not only attains a highly rewarding strategy significantly faster than state-of-the-art deep reinforcement learning algorithms, but also achieves a higher overall reward on some of the more challenging domains.