mlsanityData to 14 Oct 2026

A Clean Slate for Offline Reinforcement Learning

arXiv 2504.11453 ↗First posted 15 Apr 2025cs.LG cs.AI cs.ROcited by 11

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Progress in offline reinforcement learning (RL) has been impeded by ambiguous problem definitions and entangled algorithmic designs, resulting in inconsistent implementations, insufficient ablations, and unfair evaluations. Although offline RL explicitly avoids environment interaction, prior methods frequently employ extensive, undocumented online evaluation for hyperparameter tuning, complicating method comparisons. Moreover, existing reference implementations differ significantly in boilerplate code, obscuring their core algorithmic contributions. We address these challenges by first introducing a rigorous taxonomy and a transparent evaluation protocol that explicitly quantifies online tuning budgets. To resolve opaque algorithmic design, we provide clean, minimalistic, single-file implementations of various model-free and model-based offline RL methods, significantly enhancing clarity and achieving substantial speed-ups. Leveraging these streamlined implementations, we propose Unifloral, a unified algorithm that encapsulates diverse prior approaches within a single, comprehensive hyperparameter space, enabling algorithm development in a shared hyperparameter space. Using Unifloral with our rigorous evaluation protocol, we develop two novel algorithms - TD3-AWR (model-free) and MoBRAC (model-based) - which substantially outperform established baselines. Our implementation is publicly available at https://github.com/EmptyJackson/unifloral.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 44 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

36 of 44 references.

Closest ancestors · 21

Same direction, newest first

  1. 2023The Generalization Gap in Offline Reinforcement Learning10 Dec
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  3. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learning16 Feb
  4. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  5. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  6. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
  9. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  11. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
9 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  9. 2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 15

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2018World Modelscited by 1,513
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  9. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  10. 2022Discovered Policy Optimisationcited by 104
  11. 2023Synthetic Experience Replaycited by 101
  12. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
3 more
  1. 2021Active Offline Policy Selectioncited by 26
  2. 2022No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RLcited by 6
  3. 2023A Strong Baseline for Batch Imitation Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 6
  2. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learningcited by 2
  3. 2026CRAX: Fast Safe Reinforcement Learning Benchmarkingcited by 2
  4. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  5. 2025Fully Offline Reinforcement Learningcited by 0
  6. 2025Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Solutionscited by 0
  7. 2026When Do We Need LLMs? A Diagnostic for Language-Driven Banditscited by 0
  8. 2026Neuro-Inspired Inverse Learning for Planning and Controlcited by 0
  9. 2026RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferencescited by 0
  10. 2026Subspace Inference Enables Efficient Active Reward Learning from Preferencescited by 0
  11. 2026A Bird's-Eye View of Iterative Reward Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  2. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  3. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  4. 2021Offline RL Without Off-Policy Evaluation
  5. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  6. 2022User-Interactive Offline Reinforcement Learning
  7. 2026Active Offline-to-Online Reinforcement Learning
  8. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model