# Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update

arXiv: https://arxiv.org/abs/1805.12375 · HTML: https://mlsanity.com/paper/1805.12375/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2018.

## Built on by (most cited first, 29 of 29)

- [Principled Exploration via Optimistic Bootstrapping and Backward Induction](https://mlsanity.com/paper/2105.06022/) (2021)
- [Generalizable Episodic Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/2103.06469/) (2021)
- [Actor Prioritized Experience Replay](https://mlsanity.com/paper/2209.00532/) (2022)
- [Topological Experience Replay](https://mlsanity.com/paper/2203.15845/) (2022)
- [Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy](https://mlsanity.com/paper/2207.12141/) (2022)
- [Model-Based Episodic Memory Induces Dynamic Hybrid Controls](https://mlsanity.com/paper/2111.02104/) (2021)
- [Episodic Self-Imitation Learning with Hindsight](https://mlsanity.com/paper/2011.13467/) (2020)
- [Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation](https://mlsanity.com/paper/2405.18688/) (2024)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [Convergence Results For Q-Learning With Experience Replay](https://mlsanity.com/paper/2112.04213/) (2021)
- [Investigating the Interplay of Prioritized Replay and Generalization](https://mlsanity.com/paper/2407.09702/) (2024)
- [Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation](https://mlsanity.com/paper/2510.08962/) (2025)
- [Event Tables for Efficient Experience Replay](https://mlsanity.com/paper/2211.00576/) (2022)
- [Sequential memory improves sample and memory efficiency in Episodic Control](https://mlsanity.com/paper/2112.14734/) (2021)
- [Neighborhood Mixup Experience Replay: Local Convex Interpolation for Improved Sample Efficiency in Continuous Control Tasks](https://mlsanity.com/paper/2205.09117/) (2022)
- [Introspective Experience Replay: Look Back When Surprised](https://mlsanity.com/paper/2206.03171/) (2022)
- [Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning](https://mlsanity.com/paper/2306.15503/) (2023)
- [Reliability-Adjusted Prioritized Experience Replay](https://mlsanity.com/paper/2506.18482/) (2025)
- [Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/2505.04619/) (2025)
- [Two-Memory Reinforcement Learning](https://mlsanity.com/paper/2304.10098/) (2023)
- [Agentic Episodic Control](https://mlsanity.com/paper/2506.01442/) (2025)
- [Delayed Rewards Calibration via Reward Empirical Sufficiency](https://mlsanity.com/paper/2102.10527/) (2021)
- [Replay For Safety](https://mlsanity.com/paper/2112.04229/) (2021)
- [Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards](https://mlsanity.com/paper/2311.00426/) (2023)
- [Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations](https://mlsanity.com/paper/2401.00162/) (2023)
- [A Tighter Convergence Proof of Reverse Experience Replay](https://mlsanity.com/paper/2408.16999/) (2024)
- [Variance Reduction Based Experience Replay for Policy Optimization](https://mlsanity.com/paper/2602.05379/) (2026)
- [Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities](https://mlsanity.com/paper/2605.05812/) (2026)
- [Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning](https://mlsanity.com/paper/2607.25123/) (2026)

## Abstract (arXiv)

We propose Episodic Backward Update (EBU) - a novel deep reinforcement learning algorithm with a direct value propagation. In contrast to the conventional use of the experience replay with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state to its previous states. Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate directly through all transitions of the sampled episode. We theoretically prove the convergence of the EBU method and experimentally demonstrate its performance in both deterministic and stochastic environments. Especially in 49 games of Atari 2600 domain, EBU achieves the same mean and median human normalized performance of DQN by using only 5% and 10% of samples, respectively.