mlsanityData to 14 Oct 2026

When Does Non-Uniform Replay Matter in Reinforcement Learning?

arXiv 2605.10236 ↗First posted 11 May 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Modern off-policy reinforcement learning algorithms often rely on simple uniform replay sampling and it remains unclear when and why non-uniform replay improves over this strong baseline. Across diverse RL settings, we show that the effectiveness of non-uniform replay is governed by three factors: replay volume, the number of replayed transitions per environment step; expected recency, how recent sampled transitions are; and the entropy of the replay sampling distribution. Our main contribution is clarifying when non-uniform replay is beneficial and providing practical guidance for replay design in modern off-policy RL. Namely, we find that non-uniform replay is most beneficial when replay volume is low, and that high-entropy sampling is important even at comparable expected recency. Motivated by these findings, we adopt a simple Truncated Geometric replay that biases sampling toward recent experience while preserving high entropy and incurring negligible computational overhead. Across large-scale parallel simulation, single-task, and multi-task settings, including three modern algorithms evaluated on five RL benchmark suites, this replay sampling strategy improves sample efficiency in low-volume regimes while remaining competitive when replay volume is high.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 37 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

27 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning29 Sep
  2. 2025Compute-Optimal Scaling for Value-Based Deep RL20 Aug
  3. 2025Reliability-Adjusted Prioritized Experience Replay23 Jun
  4. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learning21 Feb
  5. 2025Value-Based Deep RL Scales Predictably6 Feb
  6. 2025Towards General-Purpose Model-Free Reinforcement Learning27 Jan
  7. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning13 Oct
  8. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May

Other references · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2018DeepMind Control Suitecited by 956
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  10. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  11. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  12. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
7 more
  1. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 83
  2. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
  3. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 44
  4. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  5. 2019Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Pastcited by 18
  6. 2025Learning Sim-to-Real Humanoid Locomotion in 15 Minutescited by 16
  7. 2026What Matters for Simulation to Online Reinforcement Learning on Real Robotscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms
  2. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning
  3. 2025Experience Replay with Random Reshuffling
  4. 2025Variance Reduction via Resampling and Experience Replay
  5. 2024A Tighter Convergence Proof of Reverse Experience Replay
  6. 2024Higher Replay Ratio Empowers Sample-Efficient Multi-Agent Reinforcement Learning
  7. 2026WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
  8. 2024Investigating the Interplay of Prioritized Replay and Generalization