mlsanityData to 14 Oct 2026

Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning

arXiv 2607.25123 ↗First posted 27 Jul 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Experience replay remains one of the most practical and useful algorithmic tools in the deep reinforcement learning (DRL) toolbox. Aside from the limited success of prioritized replay and specialized approaches for large asynchronous systems, most DRL algorithms make use of a large, uniformly sampled recency buffer---even the size, one million, remains unchanged. Could we store less data, reduce redundancy, or more effectively chain experience together to speed up value propagation and still retain the performance of large buffers? In this paper, we investigate a simple compression approach that stores representative transitions derived from the end-points of a chain of connected n-step sequences. By curating these end-points in a smaller recency buffer, our method maintains an effective memory horizon comparable to a standard large buffer while requiring an order of magnitude less storage. Through empirical evaluation, we demonstrate that this approach prevents the systematic bias inherent in naive compression strategies and matches the performance of traditional large buffers in the Pinball environment and the Atari 2600 benchmark.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 40 of its 58 references that are in the map. First posted on arXiv 27 Jul 2026.

Builds on

40 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control3 Jul
  2. 2024Prioritized Generative Replay23 Oct
  3. 2024Streaming Deep Reinforcement Learning Finally Works18 Oct
  4. 2024Investigating the Interplay of Prioritized Replay and Generalization12 Jul
  5. 2024Augmenting Replay in World Models for Continual Reinforcement Learning30 Jan

Other references · 35

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2018Experience Replay for Continual Learningcited by 1,095
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  10. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  11. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  12. 2018Selective Experience Replay for Lifelong Learningcited by 241
23 more
  1. 2020Revisiting Fundamentals of Experience Replaycited by 175
  2. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 161
  3. 2013Risk-sensitive Reinforcement Learningcited by 130
  4. 2019When to use parametric models in reinforcement learning?cited by 118
  5. 2017A Deeper Look at Experience Replaycited by 105
  6. 2023Synthetic Experience Replaycited by 101
  7. 2023Empirical Design in Reinforcement Learningcited by 55
  8. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  9. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  10. 2016Unifying task specification in reinforcement learningcited by 41
  11. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  12. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
  13. 2018Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domainscited by 19
  14. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 18
  15. 2022Topological Experience Replaycited by 17
  16. 2022Goal-Space Planning with Subgoal Modelscited by 12
  17. 2023VA-learning as a more efficient alternative to Q-learningcited by 11
  18. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 10
  19. 2022Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidationcited by 9
  20. 2023Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imagingcited by 7
  21. 2022Introspective Experience Replay: Look Back When Surprisedcited by 3
  22. 2024ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Controlcited by 2
  23. 2026Actor-Critic Algorithm for Dynamic Expectile and CVaRcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
  2. 2025Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound
  3. 2024A Tighter Convergence Proof of Reverse Experience Replay
  4. 2025Enhancing Deep Deterministic Policy Gradients on Continuous Control Tasks with Decoupled Prioritized Experience Replay
  5. 2024CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms
  6. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning
  7. 2026Uncertainty-Driven Replay Memory for Reinforcement Learning
  8. 2025Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning