# Deep Belief Markov Models for POMDP Inference

arXiv: https://arxiv.org/abs/2503.13438 · HTML: https://mlsanity.com/paper/2503.13438/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 17 Mar 2025.

## Builds on (most cited first, 27 of 27)

- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Dynamical Variational Autoencoders: A Comprehensive Review](https://mlsanity.com/paper/2008.12595/) (2020)
- [Deep Variational Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1806.02426/) (2018)
- [Learning Stochastic Recurrent Networks](https://mlsanity.com/paper/1411.7610/) (2014)
- [QMDP-Net: Deep Learning for Planning under Partial Observability](https://mlsanity.com/paper/1703.06692/) (2017)
- [Online Planning Algorithms for POMDPs](https://mlsanity.com/paper/1401.3436/) (2014)
- [AttDMM: An Attentive Deep Markov Model for Risk Scoring in Intensive Care Units](https://mlsanity.com/paper/2102.04702/) (2021)
- [POMDP inference and robust solution via deep reinforcement learning: An application to railway optimal maintenance](https://mlsanity.com/paper/2307.08082/) (2023)
- [Physics-guided Deep Markov Models for Learning Nonlinear Dynamical Systems with Uncertainty](https://mlsanity.com/paper/2110.08607/) (2021)
- [Bridging POMDPs and Bayesian decision making for robust maintenance planning under model uncertainty: An application to railway systems](https://mlsanity.com/paper/2212.07933/) (2022)
- [Robust Reinforcement Learning in POMDPs with Incomplete and Noisy Observations](https://mlsanity.com/paper/1902.05795/) (2019)
- [A multimodal dynamical variational autoencoder for audiovisual speech representation learning](https://mlsanity.com/paper/2305.03582/) (2023)
- [Reinforced Deep Markov Models With Applications in Automatic Trading](https://mlsanity.com/paper/2011.04391/) (2020)
- [Which Model to Trust: Assessing the Influence of Models on the Performance of Reinforcement Learning Algorithms for Continuous Control Tasks](https://mlsanity.com/paper/2110.13079/) (2021)

## Built on by (most cited first, 2 of 2)

- [Mathematical methods of reinforcement learning](https://mlsanity.com/paper/2607.06935/) (2026)
- [Graph-Based Inference and Topology-Aware Multi-Agent Reinforcement Learning for Large-Scale Railway Network Management](https://mlsanity.com/paper/2609.30150/) (2026)

## Abstract (arXiv)

This work introduces a novel deep learning-based architecture, termed the Deep Belief Markov Model (DBMM), which provides efficient, model-formulation agnostic inference in Partially Observable Markov Decision Process (POMDP) problems. The POMDP framework allows for modeling and solving sequential decision-making problems under observation uncertainty. In complex, high-dimensional, partially observable environments, existing methods for inference based on exact computations (e.g., via Bayes' theorem) or sampling algorithms do not scale well. Furthermore, ground truth states may not be available for learning the exact transition dynamics. DBMMs extend deep Markov models into the partially observable decision-making framework and allow efficient belief inference entirely based on available observation data via variational inference methods. By leveraging the potency of neural networks, DBMMs can infer and simulate non-linear relationships in the system dynamics and naturally scale to problems with high dimensionality and discrete or continuous variables. In addition, neural network parameters can be dynamically updated efficiently based on data availability. DBMMs can thus be used to infer a belief variable, thus enabling the derivation of POMDP solutions over the belief space. We evaluate the efficacy of the proposed methodology by evaluating the capability of model-formulation agnostic inference of DBMMs in benchmark problems that include discrete and continuous variables.