mlsanityData to 14 Oct 2026

Fully Offline Reinforcement Learning

arXiv 2505.22442 ↗First posted 28 May 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 83 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

17 of 83 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025A Clean Slate for Offline Reinforcement Learning15 Apr
  2. 2024Reinforcement Learning: An Overview6 Dec
  3. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning15 Oct
  4. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  7. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 9

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  3. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  4. 2019Batch Policy Learning under Constraintscited by 244
  5. 2023A Tutorial on Meta-Reinforcement Learningcited by 108
  6. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  7. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  8. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  9. 2023A Strong Baseline for Batch Imitation Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021A Minimalist Approach to Offline Reinforcement Learning
  2. 2020MOReL : Model-Based Offline Reinforcement Learning
  3. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  4. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
  5. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  6. 2026Active Offline-to-Online Reinforcement Learning
  7. 2020MOPO: Model-based Offline Policy Optimization
  8. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning