mlsanityData to 7 Oct 2026

Model-Based Uncertainty in Value Functions

arXiv 2302.12526 ↗First posted 24 Feb 2023cs.LG cs.AI stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over MDPs. Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation, but the over-approximation may result in inefficient exploration. We propose a new uncertainty Bellman equation whose solution converges to the true posterior variance over values and explicitly characterizes the gap in previous work. Moreover, our uncertainty quantification technique is easily integrated into common exploration strategies and scales naturally beyond the tabular setting by using standard deep reinforcement learning architectures. Experiments in difficult exploration tasks, both in tabular and continuous control settings, show that our sharper uncertainty estimates improve sample-efficiency.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 24 Feb 2023.

Builds on

24 of 39 references.

References · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  8. 2016Deep Exploration via Bootstrapped DQNcited by 518
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  10. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  11. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  12. 2017Deep Exploration via Randomized Value Functionscited by 155
12 more
  1. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
  2. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  3. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  4. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  5. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  6. 2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
  7. 2020Making Sense of Reinforcement Learning and Probabilistic Inferencecited by 30
  8. 2020Model-based Reinforcement Learning for Continuous Control with Posterior Samplingcited by 21
  9. 2021Bayesian Bellman Operatorscited by 16
  10. 2019Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansioncited by 13
  11. 2019Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimizationcited by 11
  12. 2021On-Policy Model Errors in Reinforcement Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 4; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
  2. 2023Probabilistic Inference in Reinforcement Learning Done Rightcited by 11
  3. 2025When to retrain a machine learning modelcited by 8
  4. 2023Value-Distributional Model-Based Reinforcement Learningcited by 5
  5. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  6. 2024Deep Exploration with PAC-Bayescited by 4
  7. 2023Bayesian Exploration Networkscited by 3
  8. 2023Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimizationcited by 3
  9. 2024Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learningcited by 2
  10. 2024Isoperimetry is All We Need: Langevin Posterior Sampling for RL with Sublinear Regretcited by 1
  11. 2025Universal Value-Function Uncertaintiescited by 1
  12. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0
3 more
  1. 2026Neural Nonmyopic Bayesian Optimization in Dynamic Cost Settingscited by 0
  2. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  3. 2026Deep Epistemic Value Functions for Optimistic Explorationcited by 0