mlsanityData to 7 Oct 2026

Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation

arXiv 2201.01666 ↗First posted 5 Jan 2022cs.LG cs.AI cs.ROcited by 26

Not on the map. Why?

Abstract (arXiv)

In model-free deep reinforcement learning (RL) algorithms, using noisy value estimates to supervise policy evaluation and optimization is detrimental to the sample efficiency. As this noise is heteroscedastic, its effects can be mitigated using uncertainty-based weights in the optimization process. Previous methods rely on sampled ensembles, which do not capture all aspects of uncertainty. We provide a systematic analysis of the sources of uncertainty in the noisy supervision that occurs in RL, and introduce inverse-variance RL, a Bayesian framework which combines probabilistic ensembles and Batch Inverse Variance weighting. We propose a method whereby two complementary uncertainty estimation methods account for both the Q-value and the environment stochasticity to better mitigate the negative impacts of noisy supervision. Our results show significant improvement in terms of sample efficiency on discrete and continuous control tasks.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 5 Jan 2022.

Builds on

37 of 52 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  11. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
25 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2017Deep Reinforcement Learning that Matterscited by 727
  3. 2016Deep Exploration via Bootstrapped DQNcited by 518
  4. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  5. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  6. 2019Challenges of Real-World Reinforcement Learningcited by 282
  7. 2017Noisy Networks for Explorationcited by 262
  8. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  9. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  10. 2017Deep Exploration via Randomized Value Functionscited by 155
  11. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  12. 2019Behaviour Suite for Reinforcement Learningcited by 129
  13. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  14. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  15. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 102
  16. 2019Evaluating and Calibrating Uncertainty Prediction in Regression Taskscited by 85
  17. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  18. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 68
  19. 2021Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Reviewcited by 34
  20. 2021DriverGym: Democratising Reinforcement Learning for Autonomous Drivingcited by 23
  21. 2018Reward Estimation for Variance Reduction in Deep Reinforcement Learningcited by 17
  22. 2020Temporal Difference Uncertainties as a Signal for Explorationcited by 15
  23. 2021State-Aware Variational Thompson Sampling for Deep Q-Networkscited by 5
  24. 2021Batch Inverse-Variance Weighting: Deep Heteroscedastic Regressioncited by 3
  25. 2021f-Cal: Calibrated aleatoric uncertainty estimation from neural networks for robot perceptioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 3; 2024: 8; 2025: 5; 2026: 3.

26 papers.

Sort:

By area:

  1. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
  2. 2022Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficientcited by 12
  3. 2022DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learningcited by 8
  4. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
  5. 2023Optimal Horizon-Free Reward-Free Exploration for Linear Mixture MDPscited by 7
  6. 2024Learning to Play Atari in a World of Tokenscited by 6
  7. 2025Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimationcited by 6
  8. 2024Uncertainty-Aware Reward-Free Exploration with General Function Approximationcited by 5
  9. 2025Uncertainty Prioritized Experience Replaycited by 5
  10. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
  11. 2022Nuclear Norm Maximization Based Curiosity-Driven Learningcited by 4
  12. 2024Contextualized Hybrid Ensemble Q-learning: Learning Fast with Control Priorscited by 4
14 more
  1. 2025SimBank: from Simulation to Solution in Prescriptive Process Monitoringcited by 4
  2. 2024A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertaintycited by 2
  3. 2024Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scalescited by 2
  4. 2024Active Reinforcement Learning Strategies for Offline Policy Improvementcited by 2
  5. 2026SHAPO: Sharpness-Aware Policy Optimization for Safe Explorationcited by 2
  6. 2022Normality-Guided Distributional Reinforcement Learning for Continuous Controlcited by 1
  7. 2024Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learningcited by 1
  8. 2026CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learningcited by 1
  9. 2022Dynamic Memory-based Curiosity: A Bootstrap Approach for Explorationcited by 0
  10. 2022MEET: A Monte Carlo Exploration-Exploitation Trade-off for Buffer Samplingcited by 0
  11. 2023An Architecture for Deploying Reinforcement Learning in Industrial Environmentscited by 0
  12. 2024Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learningcited by 0
  13. 2025SrSv: Integrating Sequential Rollouts with Sequential Value Estimation for Multi-agent Reinforcement Learningcited by 0
  14. 2026EasyPPO: Stabilizing the Critic Is Keycited by 0