# Generalized Bayesian deep reinforcement learning

arXiv: https://arxiv.org/abs/2412.11743 · HTML: https://mlsanity.com/paper/2412.11743/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 23 of its 67 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [(More) Efficient Reinforcement Learning via Posterior Sampling](https://mlsanity.com/paper/1306.0940/) (2013)
- [A General Framework for Updating Belief Distributions](https://mlsanity.com/paper/1306.6430/) (2013)
- [A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning](https://mlsanity.com/paper/2006.06224/) (2020)
- [Gaussian Processes for Data-Efficient Learning in Robotics and Control](https://mlsanity.com/paper/1502.02860/) (2015)
- [A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization](https://mlsanity.com/paper/1905.01332/) (2019)
- [Learning Unknown Markov Decision Processes: A Thompson Sampling Approach](https://mlsanity.com/paper/1709.04570/) (2017)
- [Assigning a value to a power likelihood in a general Bayesian model](https://mlsanity.com/paper/1701.08515/) (2017)
- [Probabilistic Forecasting with Generative Networks via Scoring Rule Minimization](https://mlsanity.com/paper/2112.08217/) (2021)
- [Survey on reinforcement learning for language processing](https://mlsanity.com/paper/2104.05565/) (2021)
- [Bayesian Synthetic Likelihood](https://mlsanity.com/paper/2305.05120/) (2023)
- [PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals](https://mlsanity.com/paper/2006.00900/) (2020)
- [Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization](https://mlsanity.com/paper/1512.07962/) (2015)
- [Objective Bayesian inference with proper scoring rules](https://mlsanity.com/paper/1711.10819/) (2017)
- [Generalized Bayesian Likelihood-Free Inference](https://mlsanity.com/paper/2104.03889/) (2021)
- [On the Asymptotic Efficiency of Approximate Bayesian Computation Estimators](https://mlsanity.com/paper/1506.03481/) (2015)
- [ABC Reinforcement Learning](https://mlsanity.com/paper/1303.6977/) (2013)

## Built on by (most cited first, 2 of 2)

- [Financial Decision Making using Reinforcement Learning with Dirichlet Priors and Quantum-Inspired Genetic Optimization](https://mlsanity.com/paper/2509.00095/) (2025)
- [Prequential posteriors](https://mlsanity.com/paper/2511.17721/) (2025)

## Abstract (arXiv)

Bayesian reinforcement learning (BRL) is a method that merges principles from Bayesian statistics and reinforcement learning to make optimal decisions in uncertain environments. As a model-based RL method, it has two key components: (1) inferring the posterior distribution of the model for the data-generating process (DGP) and (2) policy learning using the learned posterior. We propose to model the dynamics of the unknown environment through deep generative models, assuming Markov dependence. In the absence of likelihood functions for these models, we train them by learning a generalized predictive-sequential (or prequential) scoring rule (SR) posterior. We used sequential Monte Carlo (SMC) samplers to draw samples from this generalized Bayesian posterior distribution. In conjunction, to achieve scalability in the high-dimensional parameter space of the neural networks, we use the gradient-based Markov kernels within SMC. To justify the use of the prequential scoring rule posterior, we prove a Bernstein-von Mises-type theorem. For policy learning, we propose expected Thompson sampling (ETS) to learn the optimal policy by maximising the expected value function with respect to the posterior distribution. This improves upon traditional Thompson sampling (TS) and its extensions, which utilize only one sample drawn from the posterior distribution. This improvement is studied both theoretically and using simulation studies, assuming a discrete action space. Finally, we successfully extended our setup for a challenging problem with a continuous action space without theoretical guarantees.