# BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximations

arXiv: https://arxiv.org/abs/2306.00249 · HTML: https://mlsanity.com/paper/2306.00249/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 71 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges](https://mlsanity.com/paper/2110.14051/) (2021)
- [Deep Variational Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1806.02426/) (2018)
- [DESPOT: Online POMDP Planning with Regularization](https://mlsanity.com/paper/1609.03250/) (2016)
- [Partially Observable Markov Decision Processes in Robotics: A Survey](https://mlsanity.com/paper/2209.10342/) (2022)
- [Online algorithms for POMDPs with continuous state, action, and observation spaces](https://mlsanity.com/paper/1709.06196/) (2017)
- [Learning and Planning in Complex Action Spaces](https://mlsanity.com/paper/2104.06303/) (2021)
- [Approximate information state for approximate planning and reinforcement learning in partially observed systems](https://mlsanity.com/paper/2010.08843/) (2020)
- [Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving](https://mlsanity.com/paper/1905.02680/) (2019)
- [Vector Quantized Models for Planning](https://mlsanity.com/paper/2106.04615/) (2021)
- [Return-based Scaling: Yet Another Normalisation Trick for Deep RL](https://mlsanity.com/paper/2105.05347/) (2021)
- [Optimality Guarantees for Particle Belief Approximation of POMDPs](https://mlsanity.com/paper/2210.05015/) (2022)
- [HyP-DESPOT: A Hybrid Parallel Algorithm for Online Planning under Uncertainty](https://mlsanity.com/paper/1802.06215/) (2018)
- [A0C: Alpha Zero in Continuous Action Space](https://mlsanity.com/paper/1805.09613/) (2018)
- [Flow-based Recurrent Belief State Learning for POMDPs](https://mlsanity.com/paper/2205.11051/) (2022)
- [Closing the Planning-Learning Loop with Application to Autonomous Driving](https://mlsanity.com/paper/2101.03834/) (2021)
- [Improving Automated Driving through POMDP Planning with Human Internal States](https://mlsanity.com/paper/2005.14549/) (2020)
- [Learning Logic Specifications for Soft Policy Guidance in POMCP](https://mlsanity.com/paper/2303.09172/) (2023)
- [A POMDP Model for Safe Geological Carbon Sequestration](https://mlsanity.com/paper/2212.00669/) (2022)
- [Adaptive Stress Testing of Trajectory Predictions in Flight Management Systems](https://mlsanity.com/paper/2011.02559/) (2020)
- [Improved POMDP Tree Search Planning with Prioritized Action Branching](https://mlsanity.com/paper/2010.03599/) (2020)
- [Optimizing Carbon Storage Operations for Long-Term Safety](https://mlsanity.com/paper/2304.09352/) (2023)

## Built on by (most cited first, 14 of 14)

- [ConstrainedZero: Chance-Constrained POMDP Planning using Learned Probabilistic Failure Surrogates and Adaptive Safety Constraints](https://mlsanity.com/paper/2405.00644/) (2024)
- [Learning Online Belief Prediction for Efficient POMDP Planning in Autonomous Driving](https://mlsanity.com/paper/2401.15315/) (2024)
- [Constrained Hierarchical Monte Carlo Belief-State Planning](https://mlsanity.com/paper/2310.20054/) (2023)
- [Quantum Bayesian Networks Can Speed up Reinforcement Learning in Partially Observable Environments](https://mlsanity.com/paper/2507.18606/) (2025)
- [Action-Gradient Monte Carlo Tree Search for Non-Parametric Continuous (PO)MDPs](https://mlsanity.com/paper/2503.12181/) (2025)
- [GammaZero: Learning To Guide POMDP Belief Space Search With Graph Representations](https://mlsanity.com/paper/2510.14035/) (2025)
- [Neural Value Iteration](https://mlsanity.com/paper/2511.08825/) (2025)
- [POMDPPlanners: Open-Source Package for POMDP Planning](https://mlsanity.com/paper/2602.20810/) (2026)
- [SkyNet: Belief-Aware Planning for Partially-Observable Stochastic Games](https://mlsanity.com/paper/2603.27751/) (2026)
- [Adaptive mine planning under geological uncertainty: A POMDP framework for sequential decision-making](https://mlsanity.com/paper/2605.13702/) (2026)
- [VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents](https://mlsanity.com/paper/2606.19729/) (2026)
- [REBA: A Revealed Belief Automaton Framework for Online Planning in Continuous POMDPs](https://mlsanity.com/paper/2606.21971/) (2026)
- [Adaptive Mixing of Policies from Searching and Policies from Learning](https://mlsanity.com/paper/2608.15700/) (2026)
- [Belief Embedding Tree Search](https://mlsanity.com/paper/2610.15046/) (2026)

## Abstract (arXiv)

Real-world planning problems, including autonomous driving and sustainable energy applications like carbon storage and resource exploration, have recently been modeled as partially observable Markov decision processes (POMDPs) and solved using approximate methods. To solve high-dimensional POMDPs in practice, state-of-the-art methods use online planning with problem-specific heuristics to reduce planning horizons and make the problems tractable. Algorithms that learn approximations to replace heuristics have recently found success in large-scale fully observable domains. The key insight is the combination of online Monte Carlo tree search with offline neural network approximations of the optimal policy and value function. In this work, we bring this insight to partially observable domains and propose BetaZero, a belief-state planning algorithm for high-dimensional POMDPs. BetaZero learns offline approximations that replace heuristics to enable online decision making in long-horizon problems. We address several challenges inherent in large-scale partially observable domains; namely challenges of transitioning in stochastic environments, prioritizing action branching with a limited search budget, and representing beliefs as input to the network. To formalize the use of all limited search information, we train against a novel Q-weighted visit counts policy. We test BetaZero on various well-established POMDP benchmarks found in the literature and a real-world problem of critical mineral exploration. Experiments show that BetaZero outperforms state-of-the-art POMDP solvers on a variety of tasks.