mlsanityData to 7 Oct 2026

Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective

arXiv 2502.11480 ↗First posted 17 Feb 2025cs.LG stat.MLcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline model-based reinforcement learning (MBRL) serves as a competitive framework that can learn well-performing policies solely from pre-collected data with the help of learned dynamics models. To fully unleash the power of offline MBRL, model selection plays a pivotal role in determining the dynamics model utilized for downstream policy learning. However, offline MBRL conventionally relies on validation or off-policy evaluation, which are rather inaccurate due to the inherent distribution shift in offline RL. To tackle this, we propose BOMS, an active model selection framework that enhances model selection in offline MBRL with only a small online interaction budget, through the lens of Bayesian optimization (BO). Specifically, we recast model selection as BO and enable probabilistic inference in BOMS by proposing a novel model-induced kernel, which is theoretically grounded and computationally efficient. Through extensive experiments, we show that BOMS improves over the baseline methods with a small amount of online interaction comparable to only 1%-2.5% of offline training data on various RL tasks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 60 of its 66 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

60 of 66 references.

Closest ancestors · 24

Same direction, newest first

  1. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning12 Dec
  2. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  3. 2023Adversarial Model for Offline Reinforcement Learning21 Feb
  4. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  5. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning25 Oct
  6. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  7. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  11. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
12 more
  1. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  2. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  5. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  6. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 36

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  12. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
24 more
  1. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  2. 2019Batch Policy Learning under Constraintscited by 242
  3. 2017Max-value Entropy Search for Efficient Bayesian Optimizationcited by 202
  4. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  5. 2021Reinforcement learning based recommender systems: A surveycited by 187
  6. 2013Bayesian Optimization in a Billion Dimensions via Random Embeddingscited by 178
  7. 2020Model-Based Offline Planningcited by 126
  8. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  9. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  10. 2022Jump-Start Reinforcement Learningcited by 102
  11. 2019Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimizationcited by 70
  12. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  13. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  14. 2022Joint Entropy Search for Maximally-Informed Bayesian Optimizationcited by 55
  15. 2020Multi-Fidelity Bayesian Optimization via Deep Neural Networkscited by 46
  16. 2019DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systemscited by 44
  17. 2016A Reinforcement Learning System to Encourage Physical Activity in Diabetes Patientscited by 38
  18. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
  19. 2021Active Offline Policy Selectioncited by 26
  20. 2021Formalizing Generalization and Robustness of Neural Networks to Weight Perturbationscited by 22
  21. 2021Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimizationcited by 22
  22. 2022Provably Safe Deep Reinforcement Learning for Robotic Manipulation in Human Environmentscited by 16
  23. 2023The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithmscited by 12
  24. 2022Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Visioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLandercited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  2. 2025Fully Offline Reinforcement Learning
  3. 2026Active Offline-to-Online Reinforcement Learning
  4. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  5. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
  6. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning
  7. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  8. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization