mlsanityData to 7 Oct 2026

Bayesian Bellman Operators

arXiv 2106.05012 ↗First posted 9 Jun 2021cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We introduce a novel perspective on Bayesian reinforcement learning (RL); whereas existing approaches infer a posterior over the transition distribution or Q-function, we characterise the uncertainty in the Bellman operator. Our Bayesian Bellman operator (BBO) framework is motivated by the insight that when bootstrapping is introduced, model-free approaches actually infer a posterior over Bellman operators, not value functions. In this paper, we use BBO to provide a rigorous theoretical analysis of model-free Bayesian RL to better understand its relationshipto established frequentist RL methodologies. We prove that Bayesian solutions are consistent with frequentist RL solutions, even when approximate inference isused, and derive conditions for which convergence properties hold. Empirically, we demonstrate that algorithms derived from the BBO framework have sophisticated deep exploration properties that enable them to solve continuous control tasks at which state-of-the-art regularised actor-critic algorithms fail catastrophically

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 29 of its 91 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

29 of 91 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2015Variational Inference with Normalizing Flowscited by 2,161
  7. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2018DeepMind Control Suitecited by 935
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  11. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  12. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
17 more
  1. 2017Noisy Networks for Explorationcited by 262
  2. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  3. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  4. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  5. 2017Deep Exploration via Randomized Value Functionscited by 155
  6. 2017A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problemcited by 123
  7. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  8. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  9. 2017Stochastic Variance Reduction Methods for Policy Evaluationcited by 56
  10. 2018Market Making via Reinforcement Learningcited by 45
  11. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  12. 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
  13. 2017BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systemscited by 30
  14. 2018Randomized Value Functions via Multiplicative Normalizing Flowscited by 15
  15. 2019Geometric Insights into the Convergence of Nonlinear TD Learningcited by 12
  16. 2015Two Timescale Stochastic Approximation with Controlled Markov noise and Off-policy temporal difference learningcited by 8
  17. 2018Fourier Policy Gradientscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 5; 2025: 2; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  2. 2022An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Contextcited by 15
  3. 2023Model-Based Uncertainty in Value Functionscited by 15
  4. 2023Why Target Networks Stabilise Temporal Difference Methodscited by 13
  5. 2023Accelerating Value Iteration with Anchoringcited by 10
  6. 2023Diverse Projection Ensembles for Distributional Reinforcement Learningcited by 8
  7. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
  8. 2024Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learningcited by 6
  9. 2024Deep Exploration with PAC-Bayescited by 4
  10. 2023Bayesian Exploration Networkscited by 3
  11. 2023Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimizationcited by 3
  12. 2023Parameterized Projected Bellman Operatorcited by 3
4 more
  1. 2024Isoperimetry is All We Need: Langevin Posterior Sampling for RL with Sublinear Regretcited by 1
  2. 2025Universal Value-Function Uncertaintiescited by 1
  3. 2025Concurrent Learning with Aggregated States via Randomized Least Squares Value Iterationcited by 0
  4. 2026Bayesian learning for the stochastic shortest path problemcited by 0