# Fast deep reinforcement learning using online adjustments from the past

arXiv: https://arxiv.org/abs/1810.08163 · HTML: https://mlsanity.com/paper/1810.08163/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2018.

## Built on by (most cited first, 14 of 14)

- [Retrieval-Augmented Reinforcement Learning](https://mlsanity.com/paper/2202.08417/) (2022)
- [Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration](https://mlsanity.com/paper/2111.11032/) (2021)
- [Large-Scale Retrieval for Reinforcement Learning](https://mlsanity.com/paper/2206.05314/) (2022)
- [Generalizable Episodic Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/2103.06469/) (2021)
- [Model-Based Episodic Memory Induces Dynamic Hybrid Controls](https://mlsanity.com/paper/2111.02104/) (2021)
- [Neural Episodic Control with State Abstraction](https://mlsanity.com/paper/2301.11490/) (2023)
- [Sequential memory improves sample and memory efficiency in Episodic Control](https://mlsanity.com/paper/2112.14734/) (2021)
- [Revisiting Experience Replayable Conditions](https://mlsanity.com/paper/2402.10374/) (2024)
- [Memory-Augmented Graph Neural Networks: A Brain-Inspired Review](https://mlsanity.com/paper/2209.10818/) (2022)
- [RANa: Retrieval-Augmented Navigation](https://mlsanity.com/paper/2504.03524/) (2025)
- [Agentic Episodic Control](https://mlsanity.com/paper/2506.01442/) (2025)
- [Eigen Memory Trees](https://mlsanity.com/paper/2210.14077/) (2022)
- [Deep Reinforcement Learning with Task-Adaptive Retrieval via Hypernetwork](https://mlsanity.com/paper/2306.10698/) (2023)
- [Momentum Boosted Episodic Memory for Improving Learning in Long-Tailed RL Environments](https://mlsanity.com/paper/2504.05840/) (2025)

## Abstract (arXiv)

We propose Ephemeral Value Adjusments (EVA): a means of allowing deep reinforcement learning agents to rapidly adapt to experience in their replay buffer. EVA shifts the value predicted by a neural network with an estimate of the value function found by planning over experience tuples from the replay buffer near the current state. EVA combines a number of recent ideas around combining episodic memory-like structures into reinforcement learning agents: slot-based storage, content-based retrieval, and memory-based planning. We show that EVAis performant on a demonstration task and Atari games.