# Efficient exploration with Double Uncertain Value Networks

arXiv: https://arxiv.org/abs/1711.10789 · HTML: https://mlsanity.com/paper/1711.10789/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2017.

## Built on by (most cited first, 16 of 16)

- [Dropout Q-Functions for Doubly Efficient Reinforcement Learning](https://mlsanity.com/paper/2110.02034/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [A Review of Uncertainty for Deep Reinforcement Learning](https://mlsanity.com/paper/2208.09052/) (2022)
- [ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages](https://mlsanity.com/paper/2306.01460/) (2023)
- [Diverse Projection Ensembles for Distributional Reinforcement Learning](https://mlsanity.com/paper/2306.07124/) (2023)
- [World Models with Hints of Large Language Models for Goal Achieving](https://mlsanity.com/paper/2406.07381/) (2024)
- [Model-Free Active Exploration in Reinforcement Learning](https://mlsanity.com/paper/2407.00801/) (2024)
- [Echoes of Socratic Doubt: Embracing Uncertainty in Calibrated Evidential Reinforcement Learning](https://mlsanity.com/paper/2402.07107/) (2024)
- [Look Before Leap: Look-Ahead Planning with Uncertainty in Reinforcement Learning](https://mlsanity.com/paper/2503.20139/) (2025)
- [A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty](https://mlsanity.com/paper/2401.02914/) (2024)
- [ADER:Adapting between Exploration and Robustness for Actor-Critic Methods](https://mlsanity.com/paper/2109.03443/) (2021)
- [Normality-Guided Distributional Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2208.13125/) (2022)
- [Exploration with Multi-Sample Target Values for Distributional Reinforcement Learning](https://mlsanity.com/paper/2202.02693/) (2022)
- [Some Supervision Required: Incorporating Oracle Policies in Reinforcement Learning via Epistemic Uncertainty Metrics](https://mlsanity.com/paper/2208.10533/) (2022)
- [Learning What Matters: Adaptive Information-Theoretic Objectives for Robot Exploration](https://mlsanity.com/paper/2605.12084/) (2026)
- [Learning to Play: Reinforcement Learning and Games](https://mlsanity.com/paper/2610.17391/) (2026)

## Abstract (arXiv)

This paper studies directed exploration for reinforcement learning agents by tracking uncertainty about the value of each available action. We identify two sources of uncertainty that are relevant for exploration. The first originates from limited data (parametric uncertainty), while the second originates from the distribution of the returns (return uncertainty). We identify methods to learn these distributions with deep neural networks, where we estimate parametric uncertainty with Bayesian drop-out, while return uncertainty is propagated through the Bellman equation as a Gaussian distribution. Then, we identify that both can be jointly estimated in one network, which we call the Double Uncertain Value Network. The policy is directly derived from the learned distributions based on Thompson sampling. Experimental results show that both types of uncertainty may vastly improve learning in domains with a strong exploration challenge.