Universal Value-Function Uncertainties

arXiv 2505.21119 ↗First posted 27 May 2025cs.LG cs.AI stat.MLcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Estimating epistemic uncertainty in value functions is a crucial challenge for many aspects of reinforcement learning (RL), including efficient exploration, safe decision-making, and offline RL. While deep ensembles provide a robust method for quantifying value uncertainty, they come with significant computational overhead. Single-model methods, while computationally favorable, often rely on heuristics and typically require additional propagation mechanisms for myopic uncertainty estimates. In this work we introduce universal value-function uncertainties (UVU), which, similar in spirit to random network distillation (RND), quantify uncertainty as squared prediction errors between an online learner and a fixed, randomly initialized target network. Unlike RND, UVU errors reflect policy-conditional value uncertainty, incorporating the future uncertainties any given policy may encounter. This is due to the training procedure employed in UVU: the online network is trained using temporal difference learning with a synthetic reward derived from the fixed, randomly initialized target network. We provide an extensive theoretical analysis of our approach using neural tangent kernel (NTK) theory and show that in the limit of infinite network width, UVU errors are exactly equivalent to the variance of an ensemble of independent universal value functions. Empirically, we show that UVU achieves equal performance to large ensembles on challenging multi-task offline RL settings, while offering simplicity and substantial computational savings.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 69 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

49 of 69 references.

Closest ancestors · 5

Same direction, newest first

Other references · 44

2016Layer Normalizationcited by 5,579
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
2018Exploration by Random Network Distillationcited by 966
2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
2017Deep Neural Networks as Gaussian Processescited by 617
2016Deep Exploration via Bootstrapped DQNcited by 518
2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
2018Is Q-learning Provably Efficient?cited by 485

32 more

2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
2017Deep Exploration via Randomized Value Functionscited by 155
2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
2021Learning One Representation to Optimize All Rewardscited by 110
2021Repulsive Deep Ensembles are Bayesiancited by 103
2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
2017The Uncertainty Bellman Equation and Explorationcited by 81
2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
2024Simplifying Deep Temporal Difference Learningcited by 71
2018Safe Reinforcement Learning with Model Uncertainty Estimatescited by 70
2019Worst-Case Regret Bounds for Exploration via Randomized Value Functionscited by 57
2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
2023Contrastive Difference Predictive Codingcited by 35
2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
2020Optimistic Exploration even with a Pessimistic Initialisationcited by 25
2022What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?cited by 21
2021Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Drivingcited by 17
2021Bayesian Bellman Operatorscited by 16
2023Model-Based Uncertainty in Value Functionscited by 15
2024More Efficient Randomized Exploration for Reinforcement Learning via Approximate Samplingcited by 12
2019Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimizationcited by 11
2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
2023Diverse Projection Ensembles for Distributional Reinforcement Learningcited by 8
2023Exploration via Epistemic Value Estimationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

2026On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inferencecited by 0

Similar, not cited

Close in wording, no citation either way