# Compositional Learning-based Planning for Vision POMDPs

arXiv: https://arxiv.org/abs/2112.09456 · HTML: https://mlsanity.com/paper/2112.09456/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Builds on (most cited first, 30 of 33)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [Multi-Task Reinforcement Learning with Soft Modularization](https://mlsanity.com/paper/2003.13661/) (2020)
- [Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer](https://mlsanity.com/paper/1609.07088/) (2016)
- [Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning](https://mlsanity.com/paper/1711.01239/) (2017)
- [Deep Variational Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1806.02426/) (2018)
- [DESPOT: Online POMDP Planning with Regularization](https://mlsanity.com/paper/1609.03250/) (2016)
- [Online algorithms for POMDPs with continuous state, action, and observation spaces](https://mlsanity.com/paper/1709.06196/) (2017)
- [Modular Networks: Learning to Decompose Neural Computation](https://mlsanity.com/paper/1811.05249/) (2018)
- [Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors](https://mlsanity.com/paper/1805.11122/) (2018)
- [Differentiable Particle Filtering via Entropy-Regularized Optimal Transport](https://mlsanity.com/paper/2102.07850/) (2021)
- [Towards Deeper Understanding of Variational Autoencoding Models](https://mlsanity.com/paper/1702.08658/) (2017)
- [QMDP-Net: Deep Learning for Planning under Partial Observability](https://mlsanity.com/paper/1703.06692/) (2017)
- [Modular meta-learning](https://mlsanity.com/paper/1806.10166/) (2018)
- [Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules](https://mlsanity.com/paper/2006.16981/) (2020)
- [Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation](https://mlsanity.com/paper/2105.07593/) (2021)
- [Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering](https://mlsanity.com/paper/1711.00108/) (2017)
- [DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs](https://mlsanity.com/paper/1909.13003/) (2019)
- [Structured World Belief for Reinforcement Learning in POMDP](https://mlsanity.com/paper/2107.08577/) (2021)
- [Bayesian Optimized Monte Carlo Planning](https://mlsanity.com/paper/2010.03597/) (2020)
- [The Value of Inferring the Internal State of Traffic Participants for Autonomous Freeway Driving](https://mlsanity.com/paper/1702.00858/) (2017)
- [An On-Line POMDP Solver for Continuous Observation Spaces](https://mlsanity.com/paper/2011.02076/) (2020)
- [Multi-Task Learning with Sequence-Conditioned Transporter Networks](https://mlsanity.com/paper/2109.07578/) (2021)
- [Sparse tree search optimality guarantees in POMDPs with continuous observation spaces](https://mlsanity.com/paper/1910.04332/) (2019)
- [Deep Visual Navigation under Partial Observability](https://mlsanity.com/paper/2109.07752/) (2021)

## Built on by (most cited first, 8 of 8)

- [Optimality Guarantees for Particle Belief Approximation of POMDPs](https://mlsanity.com/paper/2210.05015/) (2022)
- [Sequential Monte Carlo for Policy Optimization in Continuous POMDPs](https://mlsanity.com/paper/2505.16732/) (2025)
- [VISION-SLS: Safe Perception-Based Control from Learned Visual Representations via System Level Synthesis](https://mlsanity.com/paper/2604.24894/) (2026)
- [Simplifying Complex Observation Models in Continuous POMDP Planning with Probabilistic Guarantees and Practice](https://mlsanity.com/paper/2311.07745/) (2023)
- [AgentForge: A Flexible Low-Code Platform for Reinforcement Learning Agent Design](https://mlsanity.com/paper/2410.19528/) (2024)
- [Perception-Based Beliefs for POMDPs with Visual Observations](https://mlsanity.com/paper/2602.05679/) (2026)
- [VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents](https://mlsanity.com/paper/2606.19729/) (2026)
- [Scalable Rao-Blackwellized Online Planning for High-Dimensional POMDPs](https://mlsanity.com/paper/2609.01351/) (2026)

## Abstract (arXiv)

The Partially Observable Markov Decision Process (POMDP) is a powerful framework for capturing decision-making problems that involve state and transition uncertainty. However, most current POMDP planners cannot effectively handle high-dimensional image observations prevalent in real world applications, and often require lengthy online training that requires interaction with the environment. In this work, we propose Visual Tree Search (VTS), a compositional learning and planning procedure that combines generative models learned offline with online model-based POMDP planning. The deep generative observation models evaluate the likelihood of and predict future image observations in a Monte Carlo tree search planner. We show that VTS is robust to different types of image noises that were not present during training and can adapt to different reward structures without the need to re-train. This new approach significantly and stably outperforms several baseline state-of-the-art vision POMDP algorithms while using a fraction of the training time.