mlsanityData to 7 Oct 2026

Decoupled Prioritized Resampling for Offline RL

arXiv 2306.05412 ↗First posted 8 Jun 2023cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) is challenged by the distributional shift problem. To address this problem, existing works mainly focus on designing sophisticated policy constraints between the learned policy and the behavior policy. However, these constraints are applied equally to well-performing and inferior actions through uniform sampling, which might negatively affect the learned policy. To alleviate this issue, we propose Offline Prioritized Experience Replay (OPER), featuring a class of priority functions designed to prioritize highly-rewarding transitions, making them more frequently visited during training. Through theoretical analysis, we show that this class of priority functions induce an improved behavior policy, and when constrained to this improved policy, a policy-constrained offline RL algorithm is likely to yield a better solution. We develop two practical strategies to obtain priority weights by estimating advantages based on a fitted value network (OPER-A) or utilizing trajectory returns (OPER-R) for quick computation. OPER is a plug-and-play component for offline RL algorithms. As case studies, we evaluate OPER on five different algorithms, including BC, TD3+BC, Onestep RL, CQL, and IQL. Extensive experiments demonstrate that both OPER-A and OPER-R significantly improve the performance for all baseline methods. Codes and priority weights are availiable at https://github.com/sail-sg/OPER.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 52 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

37 of 52 references.

Closest ancestors · 16

Same direction, newest first

  1. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size20 Nov
  2. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints2 Nov
  3. 2022Boosting Offline Reinforcement Learning via Data Rebalancing17 Oct
  4. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  8. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  9. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  10. 2021Offline RL Without Off-Policy Evaluation16 Jun
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
4 more
  1. 2020Critic Regularized Regression26 Jun
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 21

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2016OpenAI Gymcited by 1,684
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2015Prioritized Experience Replaycited by 1,070
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  10. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  11. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  12. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
9 more
  1. 2018Self-Imitation Learningcited by 135
  2. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  3. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  4. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  5. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  6. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  7. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  8. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  9. 2019Prioritized Sequence Experience Replaycited by 20

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 2.

6 papers.

  1. 2023Rank-DETR for High Quality Object Detectioncited by 49
  2. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  3. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  4. 2024OffLight: An Offline Multi-Agent Reinforcement Learning Framework for Traffic Signal Controlcited by 3
  5. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentationcited by 2
  6. 2025CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning
  2. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  3. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  4. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  5. 2025Sample-Efficient Policy Constraint Offline Deep Reinforcement Learning based on Sample Filtering
  6. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  7. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  8. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL