mlsanityData to 14 Oct 2026

Revisiting the Minimalist Approach to Offline Reinforcement Learning

arXiv 2305.09836 ↗First posted 16 May 2023cs.LG cs.AIcited by 126

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recent years have witnessed significant advancements in offline reinforcement learning (RL), resulting in the development of numerous algorithms with varying degrees of complexity. While these algorithms have led to noteworthy improvements, many incorporate seemingly minor design choices that impact their effectiveness beyond core algorithmic advances. However, the effect of these design choices on established baselines remains understudied. In this work, we aim to bridge this gap by conducting a retrospective analysis of recent works in offline RL and propose ReBRAC, a minimalistic algorithm that integrates such design elements built on top of the TD3+BC method. We evaluate ReBRAC on 51 datasets with both proprioceptive and visual state spaces using D4RL and V-D4RL benchmarks, demonstrating its state-of-the-art performance among ensemble-free methods in both offline and offline-to-online settings. To further illustrate the efficacy of these design choices, we perform a large-scale ablation study and hyperparameter sensitivity analysis on the scale of thousands of experiments.

Key facts. Cited by 126 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 45 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

40 of 45 references.

Closest ancestors · 22

Same direction, newest first

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  2. 2023Behavior Proximal Policy Optimization22 Feb
  3. 2023Anti-Exploration by Random Network Distillation31 Jan
  4. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  5. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size20 Nov
  6. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  7. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  8. 2022On the Role of Discount Factor in Offline Reinforcement Learning7 Jun
  9. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  10. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  11. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  12. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
10 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2016Layer Normalizationcited by 5,669
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  7. 2017Deep Reinforcement Learning that Matterscited by 754
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  9. 2017Large Batch Training of Convolutional Networkscited by 535
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  11. 2022Multi-Game Decision Transformerscited by 214
  12. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
6 more
  1. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  2. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 83
  3. 2020D2RL: Deep Dense Architectures in Reinforcement Learningcited by 42
  4. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 38
  5. 2022Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flowscited by 21
  6. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 22; 2025: 45; 2026: 56.

126 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 184
  2. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  3. 2025Reinforcement Learning with Action Chunkingcited by 107
  4. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  5. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 63
  6. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 58
  7. 2025Horizon Reduction Makes RL Scalablecited by 57
  8. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  9. 2026Q-learning with Adjoint Matchingcited by 37
  10. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 33
  11. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 31
  12. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
38 more
  1. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  2. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
  3. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  4. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 20
  5. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  6. 2025What Matters for Batch Online Reinforcement Learning in Robotics?cited by 15
  7. 2025Value Flowscited by 15
  8. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 14
  9. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  10. 2025Scalable Offline Model-Based RL with Action Chunkscited by 13
  11. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  12. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  13. 2025A Clean Slate for Offline Reinforcement Learningcited by 11
  14. 2025Yes, Q-learning Helps Offline In-Context RLcited by 10
  15. 2025Multi-agent Coordination via Flow Matchingcited by 10
  16. 2024Rethinking Optimal Transport in Offline Reinforcement Learningcited by 9
  17. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
  18. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
  19. 2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9
  20. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  21. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  22. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 6
  23. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  24. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  25. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  26. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  27. 2023Katakomba: Tools and Benchmarks for Data-Driven NetHackcited by 5
  28. 2024Bellman Diffusion Modelscited by 5
  29. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RLcited by 5
  30. 2026DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extractioncited by 5
  31. 2026Reinforcement Learning via Value Gradient Flowcited by 5
  32. 2024Simple Ingredients for Offline Reinforcement Learningcited by 4
  33. 2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
  34. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  35. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learningcited by 4
  36. 2025The Three Regimes of Offline-to-Online Reinforcement Learningcited by 4
  37. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Actioncited by 4
  38. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  2. 2026ReBRAC-v2: The Return of the King
  3. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  4. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  5. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  6. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model
  7. 2020Hyperparameter Selection for Offline Reinforcement Learning
  8. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces