# Learning in POMDPs with Monte Carlo Tree Search

arXiv: https://arxiv.org/abs/1806.05631 · HTML: https://mlsanity.com/paper/1806.05631/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2018.

## Built on by (most cited first, 15 of 15)

- [Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes](https://mlsanity.com/paper/2110.15332/) (2021)
- [Offline RL with Observation Histories: Analyzing and Improving Sample Complexity](https://mlsanity.com/paper/2310.20663/) (2023)
- [Best-Response Bayesian Reinforcement Learning with Bayes-adaptive POMDPs for Centaurs](https://mlsanity.com/paper/2204.01160/) (2022)
- [The Holy Grail of Multi-Robot Planning: Learning to Generate Online-Scalable Solutions from Offline-Optimal Experts](https://mlsanity.com/paper/2107.12254/) (2021)
- [On-Robot Bayesian Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2307.11954/) (2023)
- [Mixed-Initiative Human-Robot Teaming under Suboptimality with Online Bayesian Adaptation](https://mlsanity.com/paper/2403.16178/) (2024)
- [BADDr: Bayes-Adaptive Deep Dropout RL for POMDPs](https://mlsanity.com/paper/2202.08884/) (2022)
- [Self-Explaining Deviations for Coordination](https://mlsanity.com/paper/2207.12322/) (2022)
- [An investigation of belief-free DRL and MCTS for inspection and maintenance planning](https://mlsanity.com/paper/2312.14824/) (2023)
- [Solving Truly Massive Budgeted Monotonic POMDPs with Oracle-Guided Meta-Reinforcement Learning](https://mlsanity.com/paper/2408.07192/) (2024)
- [Rao-Blackwellized POMDP Planning](https://mlsanity.com/paper/2409.16392/) (2024)
- [Knowledge is reward: Learning optimal exploration by predictive reward cashing](https://mlsanity.com/paper/2109.08518/) (2021)
- [Sensor Control for Information Gain in Dynamic, Sparse and Partially Observed Environments](https://mlsanity.com/paper/2211.01527/) (2022)
- [Combining a Meta-Policy and Monte-Carlo Planning for Scalable Type-Based Reasoning in Partially Observable Environments](https://mlsanity.com/paper/2306.06067/) (2023)
- [Modelling crypto markets by multi-agent reinforcement learning](https://mlsanity.com/paper/2402.10803/) (2024)

## Abstract (arXiv)

The POMDP is a powerful framework for reasoning under outcome and information uncertainty, but constructing an accurate POMDP model is difficult. Bayes-Adaptive Partially Observable Markov Decision Processes (BA-POMDPs) extend POMDPs to allow the model to be learned during execution. BA-POMDPs are a Bayesian RL approach that, in principle, allows for an optimal trade-off between exploitation and exploration. Unfortunately, BA-POMDPs are currently impractical to solve for any non-trivial domain. In this paper, we extend the Monte-Carlo Tree Search method POMCP to BA-POMDPs and show that the resulting method, which we call BA-POMCP, is able to tackle problems that previous solution methods have been unable to solve. Additionally, we introduce several techniques that exploit the BA-POMDP structure to improve the efficiency of BA-POMCP along with proof of their convergence.