# Bayesian Optimized Monte Carlo Planning

arXiv: https://arxiv.org/abs/2010.03597 · HTML: https://mlsanity.com/paper/2010.03597/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [Risk-Averse Bayes-Adaptive Reinforcement Learning](https://mlsanity.com/paper/2102.05762/) (2021)
- [Optimality Guarantees for Particle Belief Approximation of POMDPs](https://mlsanity.com/paper/2210.05015/) (2022)
- [Compositional Learning-based Planning for Vision POMDPs](https://mlsanity.com/paper/2112.09456/) (2021)
- [Epistemic Monte Carlo Tree Search](https://mlsanity.com/paper/2210.13455/) (2022)
- [Adaptive Discretization using Voronoi Trees for Continuous POMDPs](https://mlsanity.com/paper/2302.10439/) (2023)
- [Online Planning for Constrained POMDPs with Continuous Spaces through Dual Ascent](https://mlsanity.com/paper/2212.12154/) (2022)
- [Uncertainty-Guided Likelihood Tree Search](https://mlsanity.com/paper/2407.03951/) (2024)
- [Adaptive Discretization using Voronoi Trees for Continuous-Action POMDPs](https://mlsanity.com/paper/2209.05733/) (2022)
- [Bayesian Optimisation Against Climate Change: Applications and Benchmarks](https://mlsanity.com/paper/2306.04343/) (2023)
- [Risk-aware Meta-level Decision Making for Exploration Under Uncertainty](https://mlsanity.com/paper/2209.05580/) (2022)
- [Constrained Hierarchical Monte Carlo Belief-State Planning](https://mlsanity.com/paper/2310.20054/) (2023)
- [A Bayesian Approach to Online Planning](https://mlsanity.com/paper/2406.02103/) (2024)
- [SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints](https://mlsanity.com/paper/2607.05363/) (2026)
- [A Surprisingly Simple Continuous-Action POMDP Solver: Lazy Cross-Entropy Search Over Policy Trees](https://mlsanity.com/paper/2305.08049/) (2023)
- [Bayesian Optimization by Kernel Regression and Density-based Exploration](https://mlsanity.com/paper/2502.06178/) (2025)
- [Action-Gradient Monte Carlo Tree Search for Non-Parametric Continuous (PO)MDPs](https://mlsanity.com/paper/2503.12181/) (2025)

## Abstract (arXiv)

Online solvers for partially observable Markov decision processes have difficulty scaling to problems with large action spaces. Monte Carlo tree search with progressive widening attempts to improve scaling by sampling from the action space to construct a policy search tree. The performance of progressive widening search is dependent upon the action sampling policy, often requiring problem-specific samplers. In this work, we present a general method for efficient action sampling based on Bayesian optimization. The proposed method uses a Gaussian process to model a belief over the action-value function and selects the action that will maximize the expected improvement in the optimal action value. We implement the proposed approach in a new online tree search algorithm called Bayesian Optimized Monte Carlo Planning (BOMCP). Several experiments show that BOMCP is better able to scale to large action space POMDPs than existing state-of-the-art tree search solvers.