mlsanityData to 14 Oct 2026

A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty

arXiv 2401.02914 ↗First posted 5 Jan 2024cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Exploration is a significant challenge in practical reinforcement learning (RL), and uncertainty-aware exploration that incorporates the quantification of epistemic and aleatory uncertainty has been recognized as an effective exploration strategy. However, capturing the combined effect of aleatory and epistemic uncertainty for decision-making is difficult. Existing works estimate aleatory and epistemic uncertainty separately and consider the composite uncertainty as an additive combination of the two. Nevertheless, the additive formulation leads to excessive risk-taking behavior, causing instability. In this paper, we propose an algorithm that clarifies the theoretical connection between aleatory and epistemic uncertainty, unifies aleatory and epistemic uncertainty estimation, and quantifies the combined effect of both uncertainties for a risk-sensitive exploration. Our method builds on a novel extension of distributional RL that estimates a parameterized return distribution whose parameters are random variables encoding epistemic uncertainty. Experimental results on tasks with exploration and risk challenges show that our method outperforms alternative approaches.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 29 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

20 of 29 references.

References · 20

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  3. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  4. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  5. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  7. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  8. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
  9. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  10. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
  11. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  12. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 40
8 more
  1. 2021How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimationcited by 27
  2. 2022Sample Efficient Deep Reinforcement Learning via Uncertainty Estimationcited by 27
  3. 2017Efficient exploration with Double Uncertain Value Networkscited by 16
  4. 2019Robust Model-free Reinforcement Learning with Multi-objective Bayesian Optimizationcited by 16
  5. 2021SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learningcited by 15
  6. 2018Exploration by Distributional Reinforcement Learningcited by 13
  7. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 10
  8. 2020MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learningcited by 6

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025Optimal Transport-Guided Safety in Temporal Difference Reinforcement Learningcited by 1
  2. 2024EX-DRL: Hedging Against Heavy Losses with EXtreme Distributional Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Deep Epistemic Value Functions for Optimistic Exploration
  2. 2025EUBRL: Epistemic Uncertainty Directed Bayesian Reinforcement Learning
  3. 2025A Novel Framework for Uncertainty-Driven Adaptive Exploration
  4. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning
  5. 2025Uncertainty Prioritized Experience Replay
  6. 2026T2PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning
  7. 2024Uncertainty-Aware Reward-Free Exploration with General Function Approximation
  8. 2025On Efficient Bayesian Exploration in Model-Based Reinforcement Learning