# Bayesian Reinforcement Learning in Factored POMDPs

arXiv: https://arxiv.org/abs/1811.05612 · HTML: https://mlsanity.com/paper/1811.05612/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2018.

## Built on by (most cited first, 15 of 15)

- [AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning](https://mlsanity.com/paper/2107.02729/) (2021)
- [Learning in Observable POMDPs, without Computationally Intractable Oracles](https://mlsanity.com/paper/2206.03446/) (2022)
- [Online Learning for Unknown Partially Observable MDPs](https://mlsanity.com/paper/2102.12661/) (2021)
- [Learning Dynamic Attribute-factored World Models for Efficient Multi-object Reinforcement Learning](https://mlsanity.com/paper/2307.09205/) (2023)
- [Decision-Making Under Uncertainty: Beyond Probabilities](https://mlsanity.com/paper/2303.05848/) (2023)
- [Provably Efficient Representation Learning with Tractable Planning in Low-Rank POMDP](https://mlsanity.com/paper/2306.12356/) (2023)
- [Sample-Efficient Learning of POMDPs with Multiple Observations In Hindsight](https://mlsanity.com/paper/2307.02884/) (2023)
- [On-Robot Bayesian Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2307.11954/) (2023)
- [Sophisticated Learning: A novel algorithm for active learning during model-based planning](https://mlsanity.com/paper/2308.08029/) (2023)
- [Towards Cost Sensitive Decision Making](https://mlsanity.com/paper/2410.03892/) (2024)
- [BADDr: Bayes-Adaptive Deep Dropout RL for POMDPs](https://mlsanity.com/paper/2202.08884/) (2022)
- [Perception-Based Beliefs for POMDPs with Visual Observations](https://mlsanity.com/paper/2602.05679/) (2026)
- [Unsupervised Neural Hidden Markov Models with a Continuous latent state space](https://mlsanity.com/paper/2106.06536/) (2021)
- [Bayesian Reinforcement Learning for Automatic Voltage Control under Cyber-Induced Uncertainty](https://mlsanity.com/paper/2305.16469/) (2023)
- [Factored Online Planning in Many-Agent POMDPs](https://mlsanity.com/paper/2312.11434/) (2023)

## Abstract (arXiv)

Bayesian approaches provide a principled solution to the exploration-exploitation trade-off in Reinforcement Learning. Typical approaches, however, either assume a fully observable environment or scale poorly. This work introduces the Factored Bayes-Adaptive POMDP model, a framework that is able to exploit the underlying structure while learning the dynamics in partially observable systems. We also present a belief tracking method to approximate the joint posterior over state and model variables, and an adaptation of the Monte-Carlo Tree Search solution method, which together are capable of solving the underlying problem near-optimally. Our method is able to learn efficiently given a known factorization or also learn the factorization and the model parameters at the same time. We demonstrate that this approach is able to outperform current methods and tackle problems that were previously infeasible.