# Variational Bayesian Reinforcement Learning with Regret Bounds

arXiv: https://arxiv.org/abs/1807.09647 · HTML: https://mlsanity.com/paper/1807.09647/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2018.

## Built on by (most cited first, 30 of 31)

- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Reward is enough for convex MDPs](https://mlsanity.com/paper/2106.00661/) (2021)
- [Reinforcement Learning, Bit by Bit](https://mlsanity.com/paper/2103.04047/) (2021)
- [MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization](https://mlsanity.com/paper/2412.12098/) (2024)
- [MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning](https://mlsanity.com/paper/2107.07184/) (2021)
- [ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs](https://mlsanity.com/paper/2302.01275/) (2023)
- [Toward Efficient Exploration by Large Language Model Agents](https://mlsanity.com/paper/2504.20997/) (2025)
- [Model-Based Uncertainty in Value Functions](https://mlsanity.com/paper/2302.12526/) (2023)
- [Probabilistic Inference in Reinforcement Learning Done Right](https://mlsanity.com/paper/2311.13294/) (2023)
- [Efficient Exploration via Epistemic-Risk-Seeking Policy Optimization](https://mlsanity.com/paper/2302.09339/) (2023)
- [Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions](https://mlsanity.com/paper/2411.00465/) (2024)
- [Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Data](https://mlsanity.com/paper/2411.03810/) (2024)
- [Variational Bayesian Optimistic Sampling](https://mlsanity.com/paper/2110.15688/) (2021)
- [Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2206.01162/) (2022)
- [On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardness](https://mlsanity.com/paper/2210.10464/) (2022)
- [IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic](https://mlsanity.com/paper/2502.19859/) (2025)
- [Factorised Active Inference for Strategic Multi-Agent Interactions](https://mlsanity.com/paper/2411.07362/) (2024)
- [Efficient Stochastic Optimal Control through Approximate Bayesian Input Inference](https://mlsanity.com/paper/2105.07693/) (2021)
- [DROMO: Distributionally Robust Offline Model-based Policy Optimization](https://mlsanity.com/paper/2109.07275/) (2021)
- [Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization](https://mlsanity.com/paper/2312.04386/) (2023)
- [Risk-sensitive control as inference with R'enyi divergence](https://mlsanity.com/paper/2411.01827/) (2024)
- [Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games](https://mlsanity.com/paper/2502.09780/) (2025)
- [Isoperimetry is All We Need: Langevin Posterior Sampling for RL with Sublinear Regret](https://mlsanity.com/paper/2412.20824/) (2024)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Adaptive Agents in Spatial Double-Auction Markets: Modeling the Emergence of Industrial Symbiosis](https://mlsanity.com/paper/2512.17979/) (2025)
- [SEREN: Knowing When to Explore and When to Exploit](https://mlsanity.com/paper/2205.15064/) (2022)
- [Dynamic mean field programming](https://mlsanity.com/paper/2206.05200/) (2022)
- [Age of Semantics in Cooperative Communications: To Expedite Simulation Towards Real via Offline Reinforcement Learning](https://mlsanity.com/paper/2209.08947/) (2022)
- [Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction](https://mlsanity.com/paper/2511.03836/) (2025)

## Abstract (arXiv)

In reinforcement learning the Q-values summarize the expected future rewards that the agent will attain. However, they cannot capture the epistemic uncertainty about those rewards. In this work we derive a new Bellman operator with associated fixed point we call the `knowledge values'. These K-values compress both the expected future rewards and the epistemic uncertainty into a single value, so that high uncertainty, high reward, or both, can yield high K-values. The key principle is to endow the agent with a risk-seeking utility function that is carefully tuned to balance exploration and exploitation. When the agent follows a Boltzmann policy over the K-values it yields a Bayes regret bound of tilde O(L √S A T), where L is the time horizon, S is the total number of states, A is the number of actions, and T is the number of elapsed timesteps. We show deep connections of this approach to the soft-max and maximum-entropy strands of research in reinforcement learning.