mlsanityData to 7 Oct 2026

Regret Minimization Experience Replay in Off-Policy Reinforcement Learning

arXiv 2105.07253 ↗First posted 15 May 2021cs.LG cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning, experience replay stores past samples for further reuse. Prioritized sampling is a promising technique to better utilize these samples. Previous criteria of prioritization include TD error, recentness and corrective feedback, which are mostly heuristically designed. In this work, we start from the regret minimization objective, and obtain an optimal prioritization strategy for Bellman update that can directly maximize the return of the policy. The theory suggests that data with higher hindsight TD error, better on-policiness and more accurate Q value should be assigned with higher weights during sampling. Thus most previous criteria only consider this strategy partially. We not only provide theoretical justifications for previous criteria, but also propose two new methods to compute the prioritization weight, namely ReMERN and ReMERT. ReMERN learns an error network, while ReMERT exploits the temporal ordering of states. Both methods outperform previous prioritized sampling algorithms in challenging RL benchmarks, including MuJoCo, Atari and Meta-World.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 15 May 2021.

Builds on

19 of 30 references.

References · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2015Prioritized Experience Replaycited by 1,070
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  8. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  9. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  10. 2020Revisiting Fundamentals of Experience Replaycited by 171
  11. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  12. 2017A Deeper Look at Experience Replaycited by 104
7 more
  1. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  2. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  3. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  4. 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
  5. 2019Prioritized Sequence Experience Replaycited by 20
  6. 2019Provably Efficient Reinforcement Learning with Aggregated Statescited by 20
  7. 2013Planning by Prioritized Sweeping with Small Backupscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 11; 2024: 5; 2025: 1; 2026: 2.

25 papers.

Sort:

By area:

  1. 2022BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasetscited by 24
  2. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  3. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 20
  4. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  5. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  6. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  7. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  8. 2023Curious Replay for Model-based Adaptationcited by 16
  9. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  10. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  11. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
  12. 2024Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theorycited by 7
13 more
  1. 2023Decoupled Prioritized Resampling for Offline RLcited by 6
  2. 2023AURO: Reinforcement Learning for Adaptive User Retention Optimization in Recommender Systemscited by 5
  3. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learningcited by 3
  4. 2023ReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learningcited by 3
  5. 2023Uncertainty-Aware Instance Reweighting for Off-Policy Learningcited by 3
  6. 2024ROER: Regularized Optimal Experience Replaycited by 3
  7. 2021Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularizationcited by 2
  8. 2022On the Reuse Bias in Off-Policy Reinforcement Learningcited by 2
  9. 2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiencescited by 1
  10. 2023Which Experiences Are Influential for Your Agent? Policy Iteration with Turn-over Dropoutcited by 0
  11. 2025Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learningcited by 0
  12. 2026VLM-Guided Experience Replaycited by 0
  13. 2026Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learningcited by 0