Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline reinforcement learning (RL) aims to optimize policies from pre-collected datasets. A bottleneck of this paradigm is managing epistemic uncertainty, which arises from limited data coverage (sample-level) and the ambiguity in identifying transition dynamics from finite data (model-level). To provide a unified quantification of these uncertainties, Bayesian RL has been proposed by treating the dynamics model as a random variable and maintaining a corresponding belief. Despite its theoretical appeal, policy optimization in Bayesian RL remains computationally challenging as it requires solving composite objectives with expectations. Prior methods either employ search-based techniques with poor computational scalability or impose restrictive posterior assumptions that sacrifice the adaptability of Bayesian RL. To address these limitations, we propose Posterior Hybrid Bayesian Belief (PhyB), which reformulates the expectation as a convex combination over a subset of dynamics models. Theoretical analysis demonstrates that the objective discrepancy induced by this approximation remains bounded. Based on PhyB, we develop an iterative regularized policy optimization algorithm that provides metric-agnostic guarantees for monotonic improvement until convergence. Empirical results demonstrate that PhyB achieves state-of-the-art performance on various benchmarks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 46 references that are in the map. First posted on arXiv 30 May 2026.
Builds on
29 of 46 references.
Closest ancestors · 19
Same direction, newest first
- 2026Offline Policy Optimization with Posterior Sampling8 May
- 2026Robust Regularized Policy Iteration under Transition Uncertainty10 Mar
- 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
- 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning23 Aug
- 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning30 Jun
- 2024Is Value Learning Really the Main Bottleneck in Offline RL?13 Jun
- 2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning27 May
- 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization11 Jan
- 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning30 Nov
- 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief13 Oct
- 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
7 more
- 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
- 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
- 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
- 2021Risk-Averse Offline Reinforcement Learning10 Feb
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
Other references · 10
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
- 2025Flow Q-Learningcited by 162
- 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
- 2022A Survey on Model-based Reinforcement Learningcited by 65
- 2019A Bayesian Approach to Robust Reinforcement Learningcited by 32
- 2021Risk-Averse Bayes-Adaptive Reinforcement Learningcited by 28
- 2019Multi-Agent Deep Reinforcement Learning for Liquidation Strategy Analysiscited by 15
- 2021Bregman Gradient Policy Optimizationcited by 15
- 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
- 2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
- 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
- 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
- 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
- 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
- 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
- 2025Fully Offline Reinforcement Learning
- 2023Bayesian Inverse Transition Learning for Offline Settings