# Partially Observable Markov Decision Processes in Robotics: A Survey

arXiv: https://arxiv.org/abs/2209.10342 · HTML: https://mlsanity.com/paper/2209.10342/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 180 references that are in the map.
- First posted on arXiv 21 Sep 2022.

## Builds on (most cited first, 30 of 38)

- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [Data-Driven Grasp Synthesis - A Survey](https://mlsanity.com/paper/1309.2660/) (2013)
- [DESPOT: Online POMDP Planning with Regularization](https://mlsanity.com/paper/1609.03250/) (2016)
- [Online algorithms for POMDPs with continuous state, action, and observation spaces](https://mlsanity.com/paper/1709.06196/) (2017)
- [Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning](https://mlsanity.com/paper/1709.10082/) (2017)
- [Online Replanning in Belief Space for Partially Observable Task and Motion Problems](https://mlsanity.com/paper/1911.04577/) (2019)
- [Online Planning Algorithms for POMDPs](https://mlsanity.com/paper/1401.3436/) (2014)
- [PORCA: Modeling and Planning for Autonomous Driving among Many Pedestrians](https://mlsanity.com/paper/1805.11833/) (2018)
- [Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes](https://mlsanity.com/paper/1302.1525/) (2013)
- [Differentiable Algorithm Networks for Composable Robot Learning](https://mlsanity.com/paper/1905.11602/) (2019)
- [HyP-DESPOT: A Hybrid Parallel Algorithm for Online Planning under Uncertainty](https://mlsanity.com/paper/1802.06215/) (2018)
- [MAGIC: Learning Macro-Actions for Online POMDP Planning](https://mlsanity.com/paper/2011.03813/) (2020)
- [The Value of Inferring the Internal State of Traffic Participants for Autonomous Freeway Driving](https://mlsanity.com/paper/1702.00858/) (2017)
- [Multiagent Rollout and Policy Iteration for POMDP with Application to Multi-Robot Repair Problems](https://mlsanity.com/paper/2011.04222/) (2020)
- [POMDP-lite for Robust Robot Planning under Uncertainty](https://mlsanity.com/paper/1602.04875/) (2016)
- [Learning Multi-Robot Decentralized Macro-Action-Based Policies via a Centralized Q-Net](https://mlsanity.com/paper/1909.08776/) (2019)
- [PLGRIM: Hierarchical Value Learning for Large-scale Exploration in Unknown Environments](https://mlsanity.com/paper/2102.05633/) (2021)
- [Decentralized Control of Partially Observable Markov Decision Processes using Belief Space Macro-actions](https://mlsanity.com/paper/1502.06030/) (2015)
- [Safe Policy Synthesis in Multi-Agent POMDPs via Discrete-Time Barrier Functions](https://mlsanity.com/paper/1903.07823/) (2019)
- [Belief-Grounded Networks for Accelerated Robot Learning under Partial Observability](https://mlsanity.com/paper/2010.09170/) (2020)
- [Robotic manipulation of multiple objects as a POMDP](https://mlsanity.com/paper/1402.0649/) (2014)
- [Multi-Resolution POMDP Planning for Multi-Object Search in 3D](https://mlsanity.com/paper/2005.02878/) (2020)
- [FIG-OP: Exploring Large-Scale Unknown Environments on a Fixed Time Budget](https://mlsanity.com/paper/2203.06316/) (2022)
- [Exploiting Submodular Value Functions For Scaling Up Active Perception](https://mlsanity.com/paper/2009.09696/) (2020)
- [Planning for robotic exploration based on forward simulation](https://mlsanity.com/paper/1502.02474/) (2015)
- [Optimal Continuous State POMDP Planning with Semantic Observations: A Variational Approach](https://mlsanity.com/paper/1807.08229/) (2018)
- [POMP: Pomcp-based Online Motion Planning for active visual search in indoor environments](https://mlsanity.com/paper/2009.08140/) (2020)

## Built on by (most cited first, 30 of 86)

- [IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks](https://mlsanity.com/paper/2506.16402/) (2025)
- [Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning](https://mlsanity.com/paper/2502.10550/) (2025)
- [Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning](https://mlsanity.com/paper/2605.06130/) (2026)
- [AVA-VLA: Improving Vision-Language-Action models with Active Visual Attention](https://mlsanity.com/paper/2511.18960/) (2025)
- [emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation](https://mlsanity.com/paper/2412.02725/) (2024)
- [A Survey on Active Feature Acquisition Strategies](https://mlsanity.com/paper/2502.11067/) (2025)
- [BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximations](https://mlsanity.com/paper/2306.00249/) (2023)
- [Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate](https://mlsanity.com/paper/2405.15384/) (2024)
- [Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learning](https://mlsanity.com/paper/2406.05064/) (2024)
- [The Wasserstein Believer: Learning Belief Updates for Partially Observable Environments through Reliable Latent Space Models](https://mlsanity.com/paper/2303.03284/) (2023)
- [AFABench: A Generic Framework for Benchmarking Active Feature Acquisition](https://mlsanity.com/paper/2508.14734/) (2025)
- [Minimax Optimal Q Learning with Nearest Neighbors](https://mlsanity.com/paper/2308.01490/) (2023)
- [Safe POMDP Online Planning via Shielding](https://mlsanity.com/paper/2309.10216/) (2023)
- [ConstrainedZero: Chance-Constrained POMDP Planning using Learned Probabilistic Failure Surrogates and Adaptive Safety Constraints](https://mlsanity.com/paper/2405.00644/) (2024)
- [PABU: Progress-Aware Belief Update for Efficient LLM Agents](https://mlsanity.com/paper/2602.09138/) (2026)
- [ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problems](https://mlsanity.com/paper/2510.07151/) (2025)
- [NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2607.06678/) (2026)
- [Fairness and Sequential Decision Making: Limits, Lessons, and Opportunities](https://mlsanity.com/paper/2301.05753/) (2023)
- [Graph-Enhanced Policy Optimization in LLM Agent Training](https://mlsanity.com/paper/2510.26270/) (2025)
- [NeuroVLA: A Brain-inspired Embodied Intelligence for Fluid and Fast Reflexive Robotics Control](https://mlsanity.com/paper/2601.14628/) (2026)
- [DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks](https://mlsanity.com/paper/2604.16484/) (2026)
- [ComTraQ-MPC: Meta-Trained DQN-MPC Integration for Trajectory Tracking with Limited Active Localization Updates](https://mlsanity.com/paper/2403.01564/) (2024)
- [Reinforcement Learning with Physics-Informed Symbolic Program Priors for Zero-Shot Wireless Indoor Navigation](https://mlsanity.com/paper/2506.22365/) (2025)
- [Improved Robustness of Deep Reinforcement Learning for Control of Time-Varying Systems by Bounded Extremum Seeking](https://mlsanity.com/paper/2510.02490/) (2025)
- [Informative Sample Selection Model for Skeleton-based Action Recognition with Limited Training Samples](https://mlsanity.com/paper/2510.25345/) (2025)
- [Learning to Look: Seeking Information for Decision Making via Policy Factorization](https://mlsanity.com/paper/2410.18964/) (2024)
- [Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review](https://mlsanity.com/paper/2505.20503/) (2025)
- [Recursively-Constrained Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2310.09688/) (2023)
- [Mixed-Initiative Human-Robot Teaming under Suboptimality with Online Bayesian Adaptation](https://mlsanity.com/paper/2403.16178/) (2024)
- [GASE: Graph Attention Sampling with Edges Fusion for Solving Vehicle Routing Problems](https://mlsanity.com/paper/2405.12475/) (2024)

## Abstract (arXiv)

Noisy sensing, imperfect control, and environment changes are defining characteristics of many real-world robot tasks. The partially observable Markov decision process (POMDP) provides a principled mathematical framework for modeling and solving robot decision and control tasks under uncertainty. Over the last decade, it has seen many successful applications, spanning localization and navigation, search and tracking, autonomous driving, multi-robot systems, manipulation, and human-robot interaction. This survey aims to bridge the gap between the development of POMDP models and algorithms at one end and application to diverse robot decision tasks at the other. It analyzes the characteristics of these tasks and connects them with the mathematical and algorithmic properties of the POMDP framework for effective modeling and solution. For practitioners, the survey provides some of the key task characteristics in deciding when and how to apply POMDPs to robot tasks successfully. For POMDP algorithm designers, the survey provides new insights into the unique challenges of applying POMDPs to robot systems and points to promising new directions for further research.