Data to 14 Oct 2026

The Uncertainty Bellman Equation and Exploration

arXiv 1709.05380 ↗First posted 15 Sep 2017cs.AI cs.LG math.OCcited by 82

Not on the map. Why?

Abstract (arXiv)

We consider the exploration/exploitation problem in reinforcement learning. For exploitation, it is well known that the Bellman equation connects the value at any time-step to the expected value at subsequent time-steps. In this paper we consider a similar textituncertainty Bellman equation (UBE), which connects the uncertainty at any time-step to the expected uncertainties at subsequent time-steps, thereby extending the potential exploratory benefit of a policy beyond individual time-steps. We prove that the unique fixed point of the UBE yields an upper bound on the variance of the posterior distribution of the Q-values induced by any policy. This bound can be much tighter than traditional count-based bonuses that compound standard deviation rather than variance. Importantly, and unlike several existing approaches to optimism, this method scales naturally to large systems with complex generalization. Substituting our UBE-exploration strategy for ε-greedy improves DQN performance on 51 out of 57 games in the Atari suite.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Sep 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 2; 2021: 15; 2022: 17; 2023: 20; 2024: 13; 2025: 10; 2026: 5.

82 papers.

Sort:

By area:

2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
2022Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoderscited by 102
2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
2021Reinforcement Learning, Bit by Bitcited by 59
2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
2023Thompson sampling for improved exploration in GFlowNetscited by 32
2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 31
2021Risk-Averse Bayes-Adaptive Reinforcement Learningcited by 29
2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 28
2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28

38 more

2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 23
2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learningcited by 19
2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
2025Toward Efficient Exploration by Large Language Model Agentscited by 17
2021Bayesian Bellman Operatorscited by 16
2022ReVar: Strengthening Policy Evaluation via Reduced Variance Samplingcited by 16
2023Posterior Sampling for Deep Reinforcement Learningcited by 16
2021The Value of Information When Deciding What to Learncited by 15
2023Model-Based Uncertainty in Value Functionscited by 15
2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14
2021Online Limited Memory Neural-Linear Bandits with Likelihood Matchingcited by 14
2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 14
2021Backprop-Free Reinforcement Learning with Active Neural Generative Codingcited by 12
2023Probabilistic Inference in Reinforcement Learning Done Rightcited by 11
2021Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Controlcited by 10
2023Efficient Exploration via Epistemic-Risk-Seeking Policy Optimizationcited by 10
2022Foundation Models for Semantic Novelty in Reinforcement Learningcited by 9
2024Efficient Multi-Policy Evaluation for Reinforcement Learningcited by 9
2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Modelscited by 8
2023Diverse Projection Ensembles for Distributional Reinforcement Learningcited by 8
2025When to retrain a machine learning modelcited by 8
2021Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulnesscited by 7
2022Epistemic Monte Carlo Tree Searchcited by 7
2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 7
2022From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Taskscited by 6
2022Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problemscited by 6
2022Autonomous Platoon Control with Integrated Deep Reinforcement Learning and Dynamic Programmingcited by 5
2022Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learningcited by 5
2022Design of experiments for the calibration of history-dependent models via deep reinforcement learning and an enhanced Kalman filtercited by 5
2023Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Controlcited by 5
2023Value-Distributional Model-Based Reinforcement Learningcited by 5
2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
2022Planning to the Information Horizon of BAMDPs via Epistemic State Abstractioncited by 4
2023STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learningcited by 4
2023Active Reinforcement Learning for Robust Building Controlcited by 4

50 most cited of 82 All 82 (JSON)