mlsanityData to 7 Oct 2026

Large Batch Experience Replay

arXiv 2110.01528 ↗First posted 4 Oct 2021cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Several algorithms have been proposed to sample non-uniformly the replay buffer of deep Reinforcement Learning (RL) agents to speed-up learning, but very few theoretical foundations of these sampling schemes have been provided. Among others, Prioritized Experience Replay appears as a hyperparameter sensitive heuristic, even though it can provide good performance. In this work, we cast the replay buffer sampling problem as an importance sampling one for estimating the gradient. This allows deriving the theoretically optimal sampling distribution, yielding the best theoretical convergence speed. Elaborating on the knowledge of the ideal sampling scheme, we exhibit new theoretical foundations of Prioritized Experience Replay. The optimal sampling distribution being intractable, we make several approximations providing good results in practice and introduce, among others, LaBER (Large Batch Experience Replay), an easy-to-code and efficient method for sampling the replay buffer. LaBER, which can be combined with Deep Q-Networks, distributional RL agents or actor-critic methods, yields improved performance over a diverse range of Atari games and PyBullet environments, compared to the base agent it is implemented on and to other prioritization schemes.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 4 Oct 2021.

Builds on

25 of 44 references.

References · 25

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2015Prioritized Experience Replaycited by 1,070
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  8. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  9. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  10. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  11. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  12. 2018Distributed Prioritized Experience Replaycited by 260
13 more
  1. 2020Revisiting Fundamentals of Experience Replaycited by 171
  2. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 132
  3. 2015Online Batch Selection for Faster Training of Neural Networkscited by 121
  4. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  5. 2017A Deeper Look at Experience Replaycited by 104
  6. 2014Stochastic Optimization with Importance Samplingcited by 91
  7. 2015Variance Reduction in SGD by Distributed Importance Samplingcited by 79
  8. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  9. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  10. 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
  11. 2019Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Pastcited by 16
  12. 2016Accelerating Deep Neural Network Training with Inconsistent Stochastic Gradient Descentcited by 6
  13. 2018State Distribution-aware Sampling for Deep Q-learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 3; 2025: 1; 2026: 3.

14 papers.

Sort:

By area:

  1. 2023Small batch deep reinforcement learningcited by 35
  2. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  3. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 20
  4. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  5. 2022Analysis of Stochastic Processes through Replay Bufferscited by 9
  6. 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learningcited by 7
  7. 2022Introspective Experience Replay: Look Back When Surprisedcited by 3
  8. 2024ROER: Regularized Optimal Experience Replaycited by 3
  9. 2023Importance Sampling for Stochastic Gradient Descent in Deep Neural Networkscited by 1
  10. 2024A Tighter Convergence Proof of Reverse Experience Replaycited by 0
  11. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  12. 2026VLM-Guided Experience Replaycited by 0
2 more
  1. 2026Correcting Within-Group Self-Selection Bias in Prioritized Replaycited by 0
  2. 2026REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reusecited by 0