mlsanityData to 7 Oct 2026

Revisiting Design Choices in Offline Model-Based Reinforcement Learning

arXiv 2110.04135 ↗First posted 8 Oct 2021cs.LG cs.AIcited by 50

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning enables agents to leverage large pre-collected datasets of environment transitions to learn control policies, circumventing the need for potentially expensive or unsafe online data collection. Significant progress has been made recently in offline model-based reinforcement learning, approaches which leverage a learned dynamics model. This typically involves constructing a probabilistic model, and using the model uncertainty to penalize rewards where there is insufficient data, solving for a pessimistic MDP that lower bounds the true MDP. Existing methods, however, exhibit a breakdown between theory and practice, whereby pessimistic return ought to be bounded by the total variation distance of the model from the true dynamics, but is instead implemented through a penalty based on estimated model uncertainty. This has spawned a variety of uncertainty heuristics, with little to no comparison between differing approaches. In this paper, we compare these heuristics, and design novel protocols to investigate their interaction with other hyperparameters, such as the number of models, or imaginary rollout horizon. Using these insights, we show that selecting these key hyperparameters using Bayesian Optimization produces superior configurations that are vastly different to those currently used in existing hand-tuned state-of-the-art methods, and result in drastically stronger performance.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 41 of its 59 references that are in the map. First posted on arXiv 8 Oct 2021.

Builds on

41 of 59 references.

Closest ancestors · 9

Same direction, newest first

  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  3. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 32

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  9. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  10. 2015Illuminating search spaces by mapping elitescited by 406
  11. 2017Population Based Training of Neural Networkscited by 351
  12. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
20 more
  1. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  2. 2020Model-Based Offline Planningcited by 126
  3. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  4. 2020Primal Wasserstein Imitation Learningcited by 120
  5. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  6. 2019A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Taskscited by 62
  7. 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 61
  8. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  9. 2019Bayesian Optimisation over Multiple Continuous and Categorical Inputscited by 55
  10. 2021Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spacescited by 51
  11. 2019Measuring the Reliability of Reinforcement Learning Algorithmscited by 49
  12. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  13. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  14. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  15. 2020Ready Policy One: World Building Through Active Learningcited by 33
  16. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  17. 2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 28
  18. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  19. 2020Model-based Policy Optimization with Unsupervised Model Adaptationcited by 16
  20. 2020Towards Tractable Optimism in Model-Based Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 12; 2024: 15; 2025: 8; 2026: 6.

50 papers.

Sort:

By area:

  1. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  2. 2023Synthetic Experience Replaycited by 100
  3. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  4. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  5. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  6. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  7. 2024Policy-Guided Diffusioncited by 40
  8. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  9. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  10. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  11. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
  12. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
38 more
  1. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
  2. 2023Model-based Offline Reinforcement Learning with Count-based Conservatismcited by 16
  3. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  4. 2023Reward-Free Curricula for Training Robust World Modelscited by 13
  5. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretrainingcited by 13
  6. 2022Adversarial Counterfactual Environment Model Learningcited by 12
  7. 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learningcited by 12
  8. 2022User-Interactive Offline Reinforcement Learningcited by 11
  9. 2023Efficient Planning with Latent Diffusioncited by 11
  10. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learningcited by 11
  11. 2023Offline Meta Reinforcement Learning with In-Distribution Online Adaptationcited by 10
  12. 2025A Clean Slate for Offline Reinforcement Learningcited by 10
  13. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  14. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learningcited by 8
  15. 2024Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?cited by 7
  16. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
  17. 2025Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lenscited by 4
  18. 2023CROP: Conservative Reward for Model-based Offline Policy Optimizationcited by 3
  19. 2025HopCast: Calibration of Autoregressive Dynamics Modelscited by 3
  20. 2024Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learningcited by 2
  21. 2024Mitigating Distribution Shift in Model-based Offline RL via Shifts-aware Reward Learningcited by 2
  22. 2025VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learningcited by 2
  23. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learningcited by 2
  24. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 2
  25. 2026Robust Regularized Policy Iteration under Transition Uncertaintycited by 2
  26. 2026The Intervention Gap in Latent World Modelscited by 2
  27. 2023HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approachcited by 1
  28. 2024Differentially Private Deep Model-Based Reinforcement Learningcited by 1
  29. 2024Offline Trajectory Optimization for Offline Reinforcement Learningcited by 1
  30. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspectivecited by 1
  31. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  32. 2026Offline Policy Optimization with Posterior Samplingcited by 1
  33. 2022Offline Policy Comparison with Confidence: Benchmarks and Baselinescited by 0
  34. 2023Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Explorationcited by 0
  35. 2024Temporal Abstraction in Reinforcement Learning with Offline Datacited by 0
  36. 2025Dual Alignment Maximin Optimization for Offline Model-based RLcited by 0
  37. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  38. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Beliefcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  2. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  3. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  4. 2023A Survey on Offline Model-Based Reinforcement Learning
  5. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  6. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  7. 2025Fully Offline Reinforcement Learning
  8. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization