mlsanityData to 13 Oct 2026

Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update

arXiv 1805.12375 ↗First posted 31 May 2018cs.LG stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

We propose Episodic Backward Update (EBU) - a novel deep reinforcement learning algorithm with a direct value propagation. In contrast to the conventional use of the experience replay with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state to its previous states. Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate directly through all transitions of the sampled episode. We theoretically prove the convergence of the EBU method and experimentally demonstrate its performance in both deterministic and stochastic environments. Especially in 49 games of Atari 2600 domain, EBU achieves the same mean and median human normalized performance of DQN by using only 5% and 10% of samples, respectively.

Key facts. Cited by 29 papers in the map (data through 13 Oct 2026). First posted on arXiv 31 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2020: 1; 2021: 8; 2022: 6; 2023: 4; 2024: 3; 2025: 4; 2026: 3.

29 papers.

Sort:

By area:

  1. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
  2. 2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 22
  3. 2022Actor Prioritized Experience Replaycited by 20
  4. 2022Topological Experience Replaycited by 17
  5. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  6. 2021Model-Based Episodic Memory Induces Dynamic Hybrid Controlscited by 14
  7. 2020Episodic Self-Imitation Learning with Hindsightcited by 10
  8. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimationcited by 9
  9. 2021Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experiencecited by 8
  10. 2021Convergence Results For Q-Learning With Experience Replaycited by 5
  11. 2024Investigating the Interplay of Prioritized Replay and Generalizationcited by 5
  12. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
17 more
  1. 2022Event Tables for Efficient Experience Replaycited by 4
  2. 2021Sequential memory improves sample and memory efficiency in Episodic Controlcited by 3
  3. 2022Neighborhood Mixup Experience Replay: Local Convex Interpolation for Improved Sample Efficiency in Continuous Control Taskscited by 3
  4. 2022Introspective Experience Replay: Look Back When Surprisedcited by 3
  5. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learningcited by 3
  6. 2025Reliability-Adjusted Prioritized Experience Replaycited by 3
  7. 2025Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulationcited by 2
  8. 2023Two-Memory Reinforcement Learningcited by 1
  9. 2025Agentic Episodic Controlcited by 1
  10. 2021Delayed Rewards Calibration via Reward Empirical Sufficiencycited by 0
  11. 2021Replay For Safetycited by 0
  12. 2023Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewardscited by 0
  13. 2023Policy Optimization with Smooth Guidance Learned from State-Only Demonstrationscited by 0
  14. 2024A Tighter Convergence Proof of Reverse Experience Replaycited by 0
  15. 2026Variance Reduction Based Experience Replay for Policy Optimizationcited by 0
  16. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalitiescited by 0
  17. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learningcited by 0