mlsanityData to 7 Oct 2026

Improving and Benchmarking Offline Reinforcement Learning Algorithms

arXiv 2306.00972 ↗First posted 1 Jun 2023cs.LGcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recently, Offline Reinforcement Learning (RL) has achieved remarkable progress with the emergence of various algorithms and datasets. However, these methods usually focus on algorithmic advancements, ignoring that many low-level implementation choices considerably influence or even drive the final performance. As a result, it becomes hard to attribute the progress in Offline RL as these choices are not sufficiently discussed and aligned in the literature. In addition, papers focusing on a dataset (e.g., D4RL) often ignore algorithms proposed on another dataset (e.g., RL Unplugged), causing isolation among the algorithms, which might slow down the overall progress. Therefore, this work aims to bridge the gaps caused by low-level choices and datasets. To this end, we empirically investigate 20 implementation choices using three representative algorithms (i.e., CQL, CRR, and IQL) and present a guidebook for choosing implementations. Following the guidebook, we find two variants CRR+ and CQL+ , achieving new state-of-the-art on D4RL. Moreover, we benchmark eight popular offline RL algorithms across datasets under unified training and evaluation framework. The findings are inspiring: the success of a learning paradigm severely depends on the data distribution, and some previous conclusions are biased by the dataset used. Our code is available at https://github.com/sail-sg/offbench.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 45 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

39 of 45 references.

Closest ancestors · 12

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  3. 2021Offline RL Without Off-Policy Evaluation16 Jun
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model13 Apr
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2020Critic Regularized Regression26 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Layer Normalizationcited by 5,579
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
15 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  2. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  3. 2018DeepMind Control Suitecited by 935
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  8. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  9. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  10. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  11. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  12. 2021Learning and Planning in Complex Action Spacescited by 76
  13. 2017Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigationcited by 74
  14. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 46
  15. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 0; 2026: 1.

6 papers.

  1. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  2. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  3. 2023Budgeting Counterfactual for Offline RLcited by 4
  4. 2024The Role of Deep Learning Regularizations on Actors in Offline RLcited by 2
  5. 2024NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulationcited by 2
  6. 2026JumpStart Your Policy Learning with Lessons from 160,000 Training Runscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning
  2. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  3. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  5. 2020Hyperparameter Selection for Offline Reinforcement Learning
  6. 2025A Clean Slate for Offline Reinforcement Learning
  7. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  8. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning