mlsanityData to 7 Oct 2026

Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

arXiv 2210.03802 ↗First posted 7 Oct 2022cs.LGcited by 16

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) addresses the problem of learning a performant policy from a fixed batch of data collected by following some behavior policy. Model-based approaches are particularly appealing in the offline setting since they can extract more learning signals from the logged dataset by learning a model of the environment. However, the performance of existing model-based approaches falls short of model-free counterparts, due to the compounding of estimation errors in the learned model. Driven by this observation, we argue that it is critical for a model-based method to understand when to trust the model and when to rely on model-free estimates, and how to act conservatively w.r.t. both. To this end, we derive an elegant and simple methodology called conservative Bayesian model-based value expansion for offline policy optimization (CBOP), that trades off model-free and model-based estimates during the policy evaluation step according to their epistemic uncertainties, and facilitates conservatism by taking a lower bound on the Bayesian posterior value estimate. On the standard D4RL continuous control tasks, we find that our method significantly outperforms previous model-based approaches: e.g., MOPO by 116.4%, MOReL by 23.2% and COMBO by 23.7%. Further, CBOP achieves state-of-the-art performance on 11 out of 18 benchmark datasets while doing on par on the remaining datasets.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 47 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

31 of 47 references.

Closest ancestors · 17

Same direction, newest first

  1. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  4. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  7. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  10. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  11. 2020Critic Regularized Regression26 Jun
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
5 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2019Batch Policy Learning under Constraintscited by 242
  11. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  12. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
2 more
  1. 2020Model-Based Offline Planningcited by 126
  2. 2019Epistemic Risk-Sensitive Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 2; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
  2. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learningcited by 17
  3. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  4. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learningcited by 11
  5. 2023How to Learn and Generalize From Three Minutes of Data: Physics-Constrained and Uncertainty-Aware Neural Stochastic Differential Equationscited by 10
  6. 2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learningcited by 10
  7. 2023Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimizationcited by 3
  8. 2024Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learningcited by 2
  9. 2024Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learningcited by 2
  10. 2024Mitigating Distribution Shift in Model-based Offline RL via Shifts-aware Reward Learningcited by 2
  11. 2024Offline Model-Based Reinforcement Learning with Anti-Explorationcited by 1
  12. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learningcited by 1
4 more
  1. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  2. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  3. 2026ReinVBC: A Model-based Reinforcement Learning Approach to Vehicle Braking Controllercited by 0
  4. 2026Neuro-Inspired Inverse Learning for Planning and Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
  2. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  3. 2026Offline Policy Optimization with Posterior Sampling
  4. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  5. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  6. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
  7. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
  8. 2023Conservative State Value Estimation for Offline Reinforcement Learning