mlsanityData to 7 Oct 2026

ROER: Regularized Optimal Experience Replay

arXiv 2407.03995 ↗First posted 4 Jul 2024cs.LG cs.AI cs.ROcited by 3

Not on the map. Why?

Abstract (arXiv)

Experience replay serves as a key component in the success of online reinforcement learning (RL). Prioritized experience replay (PER) reweights experiences by the temporal difference (TD) error empirically enhancing the performance. However, few works have explored the motivation of using TD error. In this work, we provide an alternative perspective on TD-error-based reweighting. We show the connections between the experience prioritization and occupancy optimization. By using a regularized RL objective with f-divergence regularizer and employing its dual form, we show that an optimal solution to the objective is obtained by shifting the distribution of off-policy data in the replay buffer towards the on-policy optimal distribution using TD-error-based occupancy ratios. Our derivation results in a new pipeline of TD error prioritization. We specifically explore the KL divergence as the regularizer and obtain a new form of prioritization scheme, the regularized optimal experience replay (ROER). We evaluate the proposed prioritization scheme with the Soft Actor-Critic (SAC) algorithm in continuous control MuJoCo and DM Control benchmark tasks where our proposed scheme outperforms baselines in 6 out of 11 tasks while the results of the rest match with or do not deviate far from the baselines. Further, using pretraining, ROER achieves noticeable improvement on difficult Antmaze environment where baselines fail, showing applicability to offline-to-online fine-tuning. Code is available at urlhttps://github.com/XavierChanglingLi/Regularized-Optimal-Experience-Replay.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

39 of 52 references.

References · 39

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2021Convex optimizationcited by 2,479
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2015Prioritized Experience Replaycited by 1,070
  9. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
27 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  2. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  3. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  4. 2018Maximum a Posteriori Policy Optimisationcited by 318
  5. 2019A Theory of Regularized Markov Decision Processescited by 279
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  8. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  9. 2017A unified view of entropy-regularized Markov decision processescited by 174
  10. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  11. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  12. 2017A Deeper Look at Experience Replaycited by 104
  13. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  14. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  15. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  16. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  17. 2017On the regularization of Wasserstein GANscited by 65
  18. 2019Experience Replay Optimizationcited by 47
  19. 2018Remember and Forget for Experience Replaycited by 42
  20. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  21. 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
  22. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  23. 2017f-Divergence constrained policy improvementcited by 15
  24. 2021Large Batch Experience Replaycited by 14
  25. 2022Revisiting Discrete Soft Actor-Criticcited by 11
  26. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularizationcited by 3
  27. 2021Balancing Value Underestimation and Overestimation with Realistic Actor-Criticcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026VLM-Guided Experience Replaycited by 0
  2. 2026Scaling up Energy-Aware Multi-Agent Reinforcement Learning for Mission-Oriented Drone Networks with Individual Rewardcited by 0
  3. 2026Correcting Within-Group Self-Selection Bias in Prioritized Replaycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Decoupled Prioritized Resampling for Offline RL
  2. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning
  3. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
  4. 2024Prioritized Generative Replay
  5. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learning
  6. 2025Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning
  7. 2026DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay
  8. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices