# Bayesian Policy Optimization for Model Uncertainty

arXiv: https://arxiv.org/abs/1810.01014 · HTML: https://mlsanity.com/paper/1810.01014/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2018.

## Built on by (most cited first, 12 of 12)

- [Robust Policy Learning over Multiple Uncertainty Sets](https://mlsanity.com/paper/2202.07013/) (2022)
- [Training a Resilient Q-Network against Observational Interference](https://mlsanity.com/paper/2102.09677/) (2021)
- [Reinforcement Learning in Presence of Discrete Markovian Context Evolution](https://mlsanity.com/paper/2202.06557/) (2022)
- [Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variables](https://mlsanity.com/paper/2210.12262/) (2022)
- [Meta Reinforcement Learning with Finite Training Tasks -- a Density Estimation Approach](https://mlsanity.com/paper/2206.10716/) (2022)
- [Planning to the Information Horizon of BAMDPs via Epistemic State Abstraction](https://mlsanity.com/paper/2210.16872/) (2022)
- [DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learning](https://mlsanity.com/paper/2402.15957/) (2024)
- [Model Selection for Production System via Automated Online Experiments](https://mlsanity.com/paper/2105.13420/) (2021)
- [Uncertainty Aware System Identification with Universal Policies](https://mlsanity.com/paper/2202.05844/) (2022)
- [Gated Linear Model induced U-net for surrogate modeling and uncertainty quantification](https://mlsanity.com/paper/2111.05123/) (2021)
- [Bayesian Reinforcement Learning for Automatic Voltage Control under Cyber-Induced Uncertainty](https://mlsanity.com/paper/2305.16469/) (2023)
- [Offline Bayesian Aleatoric and Epistemic Uncertainty Quantification and Posterior Value Optimisation in Finite-State MDPs](https://mlsanity.com/paper/2406.02456/) (2024)

## Abstract (arXiv)

Addressing uncertainty is critical for autonomous systems to robustly adapt to the real world. We formulate the problem of model uncertainty as a continuous Bayes-Adaptive Markov Decision Process (BAMDP), where an agent maintains a posterior distribution over latent model parameters given a history of observations and maximizes its expected long-term reward with respect to this belief distribution. Our algorithm, Bayesian Policy Optimization, builds on recent policy optimization algorithms to learn a universal policy that navigates the exploration-exploitation trade-off to maximize the Bayesian value function. To address challenges from discretizing the continuous latent parameter space, we propose a new policy network architecture that encodes the belief distribution independently from the observable state. Our method significantly outperforms algorithms that address model uncertainty without explicitly reasoning about belief distributions and is competitive with state-of-the-art Partially Observable Markov Decision Process solvers.