# On-Robot Bayesian Reinforcement Learning for POMDPs

arXiv: https://arxiv.org/abs/2307.11954 · HTML: https://mlsanity.com/paper/2307.11954/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 22 Jul 2023.

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Preparing for the Unknown: Learning a Universal Policy with Online System Identification](https://mlsanity.com/paper/1702.02453/) (2017)
- [End-to-End Robotic Reinforcement Learning without Reward Engineering](https://mlsanity.com/paper/1904.07854/) (2019)
- [A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning](https://mlsanity.com/paper/2208.07860/) (2022)
- [Sample Efficient Grasp Learning Using Equivariant Models](https://mlsanity.com/paper/2202.09468/) (2022)
- [On-Robot Learning With Equivariant Models](https://mlsanity.com/paper/2203.04923/) (2022)
- [Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations](https://mlsanity.com/paper/2002.09884/) (2020)
- [Learning in POMDPs with Monte Carlo Tree Search](https://mlsanity.com/paper/1806.05631/) (2018)
- [Bayesian Reinforcement Learning in Factored POMDPs](https://mlsanity.com/paper/1811.05612/) (2018)
- [BADDr: Bayes-Adaptive Deep Dropout RL for POMDPs](https://mlsanity.com/paper/2202.08884/) (2022)

## Built on by (most cited first, 3 of 3)

- [Belief Aided Navigation using Bayesian Reinforcement Learning for Avoiding Humans in Blind Spots](https://mlsanity.com/paper/2403.10105/) (2024)
- [Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systems](https://mlsanity.com/paper/2507.21589/) (2025)
- [Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates](https://mlsanity.com/paper/2610.14042/) (2026)

## Abstract (arXiv)

Robot learning is often difficult due to the expense of gathering data. The need for large amounts of data can, and should, be tackled with effective algorithms and leveraging expert information on robot dynamics. Bayesian reinforcement learning (BRL), thanks to its sample efficiency and ability to exploit prior knowledge, is uniquely positioned as such a solution method. Unfortunately, the application of BRL has been limited due to the difficulties of representing expert knowledge as well as solving the subsequent inference problem. This paper advances BRL for robotics by proposing a specialized framework for physical systems. In particular, we capture this knowledge in a factored representation, then demonstrate the posterior factorizes in a similar shape, and ultimately formalize the model in a Bayesian framework. We then introduce a sample-based online solution method, based on Monte-Carlo tree search and particle filtering, specialized to solve the resulting model. This approach can, for example, utilize typical low-level robot simulators and handle uncertainty over unknown dynamics of the environment. We empirically demonstrate its efficiency by performing on-robot learning in two human-robot interaction tasks with uncertainty about human behavior, achieving near-optimal performance after only a handful of real-world episodes. A video of learned policies is at https://youtu.be/H9xp60ngOes.