# Online algorithms for POMDPs with continuous state, action, and observation spaces

arXiv: https://arxiv.org/abs/1709.06196 · HTML: https://mlsanity.com/paper/1709.06196/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2017.

## Built on by (most cited first, 30 of 84)

- [Robust Reinforcement Learning on State Observations with Learned Optimal Adversary](https://mlsanity.com/paper/2101.08452/) (2021)
- [Partially Observable Markov Decision Processes in Robotics: A Survey](https://mlsanity.com/paper/2209.10342/) (2022)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning](https://mlsanity.com/paper/2402.17930/) (2024)
- [Optimality Guarantees for Particle Belief Approximation of POMDPs](https://mlsanity.com/paper/2210.05015/) (2022)
- [Safe Output Feedback Motion Planning from Images via Learned Perception Modules and Contraction Theory](https://mlsanity.com/paper/2206.06553/) (2022)
- [VJEPA: Variational Joint Embedding Predictive Architectures as Probabilistic World Models](https://mlsanity.com/paper/2601.14354/) (2026)
- [Active Uncertainty Reduction for Safe and Efficient Interaction Planning: A Shielding-Aware Dual Control Approach](https://mlsanity.com/paper/2302.00171/) (2023)
- [Adaptive Informative Path Planning Using Deep Reinforcement Learning for UAV-based Active Sensing](https://mlsanity.com/paper/2109.13570/) (2021)
- [BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximations](https://mlsanity.com/paper/2306.00249/) (2023)
- [Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning](https://mlsanity.com/paper/2410.11234/) (2024)
- [Compositional Learning-based Planning for Vision POMDPs](https://mlsanity.com/paper/2112.09456/) (2021)
- [ConstrainedZero: Chance-Constrained POMDP Planning using Learned Probabilistic Failure Surrogates and Adaptive Safety Constraints](https://mlsanity.com/paper/2405.00644/) (2024)
- [Epistemic Monte Carlo Tree Search](https://mlsanity.com/paper/2210.13455/) (2022)
- [Active teacher selection for reward learning](https://mlsanity.com/paper/2310.15288/) (2023)
- [Simplified Belief-Dependent Reward MCTS Planning with Guaranteed Tree Consistency](https://mlsanity.com/paper/2105.14239/) (2021)
- [Adaptive Information Belief Space Planning](https://mlsanity.com/paper/2201.05673/) (2022)
- [A POMDP Model for Safe Geological Carbon Sequestration](https://mlsanity.com/paper/2212.00669/) (2022)
- [Adaptive Discretization using Voronoi Trees for Continuous POMDPs](https://mlsanity.com/paper/2302.10439/) (2023)
- [Mask Atari for Deep Reinforcement Learning as POMDP Benchmarks](https://mlsanity.com/paper/2203.16777/) (2022)
- [Online Planning for Constrained POMDPs with Continuous Spaces through Dual Ascent](https://mlsanity.com/paper/2212.12154/) (2022)
- [Learning Online Belief Prediction for Efficient POMDP Planning in Autonomous Driving](https://mlsanity.com/paper/2401.15315/) (2024)
- [Monte Carlo Information-Oriented Planning](https://mlsanity.com/paper/2103.11345/) (2021)
- [Online POMDP Planning via Simplification](https://mlsanity.com/paper/2105.05296/) (2021)
- [Intention-Aware Navigation in Crowds with Extended-Space POMDP Planning](https://mlsanity.com/paper/2206.10028/) (2022)
- [Sequential Bayesian Optimization for Adaptive Informative Path Planning with Multimodal Sensing](https://mlsanity.com/paper/2209.07660/) (2022)
- [Simplified Continuous High Dimensional Belief Space Planning with Adaptive Probabilistic Belief-dependent Constraints](https://mlsanity.com/paper/2302.06697/) (2023)
- [Sequential Monte Carlo for Policy Optimization in Continuous POMDPs](https://mlsanity.com/paper/2505.16732/) (2025)
- [Risk Aware Adaptive Belief-dependent Probabilistically Constrained Continuous POMDP Planning](https://mlsanity.com/paper/2209.02679/) (2022)
- [Adaptive Discretization using Voronoi Trees for Continuous-Action POMDPs](https://mlsanity.com/paper/2209.05733/) (2022)

## Abstract (arXiv)

Online solvers for partially observable Markov decision processes have been applied to problems with large discrete state spaces, but continuous state, action, and observation spaces remain a challenge. This paper begins by investigating double progressive widening (DPW) as a solution to this challenge. However, we prove that this modification alone is not sufficient because the belief representations in the search tree collapse to a single particle causing the algorithm to converge to a policy that is suboptimal regardless of the computation time. This paper proposes and evaluates two new algorithms, POMCPOW and PFT-DPW, that overcome this deficiency by using weighted particle filtering. Simulation results show that these modifications allow the algorithms to be successful where previous approaches fail.