Sequential Monte Carlo for Policy Optimization in Continuous POMDPs
Not on the map. Why?
Abstract (arXiv)
Optimal decision-making under partial observability requires agents to balance reducing uncertainty (exploration) against pursuing immediate objectives (exploitation). In this paper, we introduce a novel policy optimization framework for continuous partially observable Markov decision processes (POMDPs) that explicitly addresses this challenge. Our method casts policy learning as probabilistic inference in a non-Markovian Feynman--Kac model that inherently captures the value of information gathering by anticipating future observations, without requiring suboptimal approximations or handcrafted heuristics. To optimize policies under this model, we develop a nested sequential Monte Carlo (SMC) algorithm that efficiently estimates a history-dependent policy gradient under samples from the optimal trajectory distribution induced by the POMDP. We demonstrate the effectiveness of our algorithm across standard continuous POMDP benchmarks, where existing methods struggle to act under uncertainty.
Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 31 of its 89 references that are in the map. First posted on arXiv 22 May 2025.
Builds on
31 of 89 references.
References · 31
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,024
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,223
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,620
- 2016Variational Inference: A Review for Statisticianscited by 1,469
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,241
- 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 477
- 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 379
- 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
- 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
- 2018Deep Variational Reinforcement Learning for POMDPscited by 123
- 2016DESPOT: Online POMDP Planning with Regularizationcited by 105
19 more
- 2017Variational Sequential Monte Carlocited by 94
- 2017Online algorithms for POMDPs with continuous state, action, and observation spacescited by 82
- 2021Differentiable Particle Filtering via Entropy-Regularized Optimal Transportcited by 59
- 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
- 2014On Particle Methods for Parameter Estimation in State-Space Modelscited by 47
- 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 46
- 2014Online Planning Algorithms for POMDPscited by 42
- 2020Discriminative Particle Filter Reinforcement Learning for Complex Partial Observationscited by 27
- 2013Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processescited by 24
- 2019DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPscited by 21
- 2021Structured World Belief for Reinforcement Learning in POMDPcited by 21
- 2021Recurrent Off-policy Baselines for Memory-based Continuous Controlcited by 19
- 2022Flow-based Recurrent Belief State Learning for POMDPscited by 16
- 2019Stochastic Optimal Control as Approximate Input Inferencecited by 14
- 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
- 2022Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processescited by 10
- 2021Compositional Learning-based Planning for Vision POMDPscited by 8
- 2022On backward smoothing algorithmscited by 7
- 2024Recursive Nested Filtering for Efficient Amortized Bayesian Experimental Designcited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.