mlsanityData to 14 Oct 2026

Flow-based Recurrent Belief State Learning for POMDPs

arXiv 2205.11051 ↗First posted 23 May 2022cs.LG stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Partially Observable Markov Decision Process (POMDP) provides a principled and generic framework to model real world sequential decision making processes but yet remains unsolved, especially for high dimensional continuous space and unknown models. The main challenge lies in how to accurately obtain the belief state, which is the probability distribution over the unobservable environment states given historical information. Accurately calculating this belief state is a precondition for obtaining an optimal policy of POMDPs. Recent advances in deep learning techniques show great potential to learn good belief states. However, existing methods can only learn approximated distribution with limited flexibility. In this paper, we introduce the FlOw-based Recurrent BElief State model (FORBES), which incorporates normalizing flows into the variational inference to learn general continuous belief states for POMDPs. Furthermore, we show that the learned belief states can be plugged into downstream RL algorithms to improve performance. In experiments, we show that our methods successfully capture the complex belief states that enable multi-modal predictions as well as high quality reconstructions, and results on challenging visual-motor control tasks show that our method achieves superior performance and sample efficiency.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 39 of its 67 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

39 of 67 references.

References · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2015Variational Inference with Normalizing Flowscited by 2,170
  6. 2016Density estimation using Real NVPcited by 2,042
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  8. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  9. 2018World Modelscited by 1,513
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  11. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  12. 2020Mastering Atari with Discrete World Modelscited by 1,007
27 more
  1. 2016Deep Variational Information Bottleneckcited by 930
  2. 2017Masked Autoregressive Flow for Density Estimationcited by 684
  3. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  4. 2015Generating Sentences from a Continuous Spacecited by 612
  5. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  6. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 262
  7. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  8. 2018Neural Autoregressive Flowscited by 208
  9. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  10. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 156
  11. 2018Deep Variational Reinforcement Learning for POMDPscited by 125
  12. 2020Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximatorscited by 85
  13. 2018Sylvester Normalizing Flows for Variational Inferencecited by 72
  14. 2019Shaping Belief States with Generative Environment Models for RLcited by 66
  15. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
  16. 2018Probabilistic Recurrent State-Space Modelscited by 64
  17. 2018Temporal Difference Variational Auto-Encodercited by 49
  18. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  19. 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 47
  20. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
  21. 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 38
  22. 2019Particle Filter Recurrent Neural Networkscited by 36
  23. 2020The Expressive Power of a Class of Normalizing Flow Modelscited by 35
  24. 2020PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inferencecited by 23
  25. 2018Randomized Value Functions via Multiplicative Normalizing Flowscited by 15
  26. 2020Bridging Imagination and Reality for Model-Based Deep Reinforcement Learningcited by 15
  27. 2018Variational Inference for Data-Efficient Model Learning in POMDPscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 3; 2024: 5; 2025: 5.

16 papers.

Sort:

By area:

  1. 2023BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximationscited by 14
  2. 2024DAG-Plan: Generating Directed Acyclic Dependency Graphs for Dual-Arm Cooperative Planningcited by 14
  3. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  4. 2023The Wasserstein Believer: Learning Belief Updates for Partially Observable Environments through Reliable Latent Space Modelscited by 9
  5. 2022EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Modelcited by 8
  6. 2024Efficient and Generalized end-to-end Autonomous Driving System with Latent Deep Reinforcement Learning and Demonstrationscited by 7
  7. 2023Statistically Efficient Bayesian Sequential Experiment Design via Reinforcement Learning with Cross-Entropy Estimatorscited by 6
  8. 2024VLASCD: A Visual Language Action Model for Simultaneous Chatting and Decision Makingcited by 5
  9. 2022CtrlFormer: Learning Transferable State Representation for Visual Control via Transformercited by 4
  10. 2024An Empirical Study on the Power of Future Prediction in Partially Observable Environmentscited by 4
  11. 2025Sequential Monte Carlo for Policy Optimization in Continuous POMDPscited by 4
  12. 2024Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbationscited by 2
4 more
  1. 2025Dream to Chat: Model-based Reinforcement Learning on Dialogues with User Belief Modelingcited by 1
  2. 2025ESCORT: Efficient Stein-variational and Sliced Consistency-Optimized Temporal Belief Representation for POMDPscited by 1
  3. 2025Success in Humanoid Reinforcement Learning under Partial Observationcited by 0
  4. 2025Predictive Coding Enhances Meta-RL To Achieve Interpretable Bayes-Optimal Belief Representation Under Partial Observabilitycited by 0