# Learning Online Belief Prediction for Efficient POMDP Planning in Autonomous Driving

arXiv: https://arxiv.org/abs/2401.15315 · HTML: https://mlsanity.com/paper/2401.15315/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving.
- Builds on 34 of its 47 references that are in the map.
- First posted on arXiv 27 Jan 2024.

## Placed in

- End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/)

## Builds on (most cited first, 30 of 34)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Monte Carlo Tree Search: A Review of Recent Modifications and Applications](https://mlsanity.com/paper/2103.04931/) (2021)
- [GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving](https://mlsanity.com/paper/2303.05760/) (2023)
- [MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying](https://mlsanity.com/paper/2306.17770/) (2023)
- [Deep Variational Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1806.02426/) (2018)
- [Online algorithms for POMDPs with continuous state, action, and observation spaces](https://mlsanity.com/paper/1709.06196/) (2017)
- [PiP: Planning-informed Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/2003.11476/) (2020)
- [Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Driving](https://mlsanity.com/paper/2207.10422/) (2022)
- [Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors](https://mlsanity.com/paper/1805.11122/) (2018)
- [QMDP-Net: Deep Learning for Planning under Partial Observability](https://mlsanity.com/paper/1703.06692/) (2017)
- [Identifying Driver Interactions via Conditional Behavior Prediction](https://mlsanity.com/paper/2104.09959/) (2021)
- [DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Driving](https://mlsanity.com/paper/2310.05885/) (2023)
- [Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving](https://mlsanity.com/paper/2212.08787/) (2022)
- [Deep Interactive Motion Prediction and Planning: Playing Games with Motion Prediction Models](https://mlsanity.com/paper/2204.02392/) (2022)
- [Bootstrap Motion Forecasting With Self-Consistent Constraints](https://mlsanity.com/paper/2204.05859/) (2022)
- [EPSILON: An Efficient Planning System for Automated Vehicles in Highly Interactive Environments](https://mlsanity.com/paper/2108.07993/) (2021)
- [Efficient Game-Theoretic Planning with Prediction Heuristic for Socially-Compliant Autonomous Driving](https://mlsanity.com/paper/2207.03673/) (2022)
- [BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximations](https://mlsanity.com/paper/2306.00249/) (2023)
- [LeTS-Drive: Driving in a Crowd by Learning from Tree Search](https://mlsanity.com/paper/1905.12197/) (2019)
- [Closing the Planning-Learning Loop with Application to Autonomous Driving](https://mlsanity.com/paper/2101.03834/) (2021)
- [Improving Automated Driving through POMDP Planning with Human Internal States](https://mlsanity.com/paper/2005.14549/) (2020)
- [MBAPPE: MCTS-Built-Around Prediction for Planning Explicitly](https://mlsanity.com/paper/2309.08452/) (2023)
- [A Taxonomy and Review of Algorithms for Modeling and Predicting Human Driver Behavior](https://mlsanity.com/paper/2006.08832/) (2020)
- [Interventional Behavior Prediction: Avoiding Overly Confident Anticipation in Interactive Prediction](https://mlsanity.com/paper/2204.08665/) (2022)

## Built on by (most cited first, 5 of 5)

- [Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agents](https://mlsanity.com/paper/2509.23141/) (2025)
- [Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning](https://mlsanity.com/paper/2410.05582/) (2024)
- [SceneMotion: From Agent-Centric Embeddings to Scene-Wide Forecasts](https://mlsanity.com/paper/2408.01537/) (2024)
- [Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges](https://mlsanity.com/paper/2512.00021/) (2025)
- [From Static to Dynamic: a Survey of Topology-Aware Perception in Autonomous Driving](https://mlsanity.com/paper/2509.23641/) (2025)

## Abstract (arXiv)

Effective decision-making in autonomous driving relies on accurate inference of other traffic agents' future behaviors. To achieve this, we propose an online belief-update-based behavior prediction model and an efficient planner for Partially Observable Markov Decision Processes (POMDPs). We develop a Transformer-based prediction model, enhanced with a recurrent neural memory model, to dynamically update latent belief state and infer the intentions of other agents. The model can also integrate the ego vehicle's intentions to reflect closed-loop interactions among agents, and it learns from both offline data and online interactions. For planning, we employ a Monte-Carlo Tree Search (MCTS) planner with macro actions, which reduces computational complexity by searching over temporally extended action steps. Inside the MCTS planner, we use predicted long-term multi-modal trajectories to approximate future updates, which eliminates iterative belief updating and improves the running efficiency. Our approach also incorporates deep Q-learning (DQN) as a search prior, which significantly improves the performance of the MCTS planner. Experimental results from simulated environments validate the effectiveness of our proposed method. The online belief update model can significantly enhance the accuracy and temporal consistency of predictions, leading to improved decision-making performance. Employing DQN as a search prior in the MCTS planner considerably boosts its performance and outperforms an imitation learning-based prior. Additionally, we show that the MCTS planning with macro actions substantially outperforms the vanilla method in terms of performance and efficiency.