mlsanityData to 7 Oct 2026

Ensemble Bootstrapping for Q-Learning

arXiv 2103.00445 ↗First posted 28 Feb 2021cs.LG cs.AI stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

Q-learning (QL), a common reinforcement learning algorithm, suffers from over-estimation bias due to the maximization term in the optimal Bellman operator. This bias may lead to sub-optimal behavior. Double-Q-learning tackles this issue by utilizing two estimators, yet results in an under-estimation bias. Similar to over-estimation in Q-learning, in certain scenarios, the under-estimation bias may degrade performance. In this work, we introduce a new bias-reduced algorithm called Ensemble Bootstrapped Q-Learning (EBQL), a natural extension of Double-Q-learning to ensembles. We analyze our method both theoretically and empirically. Theoretically, we prove that EBQL-like updates yield lower MSE when estimating the maximal mean of a set of independent random variables. Empirically, we show that there exist domains where both over and under-estimation result in sub-optimal performance. Finally, We demonstrate the superior performance of a deep RL variant of EBQL over other deep QL algorithms for a suite of ATARI games.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 15 of its 39 references that are in the map. First posted on arXiv 28 Feb 2021.

Builds on

15 of 39 references.

References · 15

  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  3. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  5. 2018Learning Dexterous In-Hand Manipulationcited by 643
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2018Is Q-learning Provably Efficient?cited by 485
  8. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  9. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  10. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  11. 2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 163
  12. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
3 more
  1. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  2. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  3. 2017Applications of Deep Learning and Reinforcement Learning to Biological Datacited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 5; 2024: 10; 2025: 7; 2026: 4.

30 papers.

Sort:

By area:

  1. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
  2. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  3. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  4. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  5. 2023Ensemble Reinforcement Learning: A Surveycited by 16
  6. 2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
  7. 2023Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedbackcited by 9
  8. 2024The Curse of Diversity in Ensemble-Based Explorationcited by 6
  9. 2023Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Rewardcited by 5
  10. 2024Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimizationcited by 5
  11. 2024Leveraging Digital Cousins for Ensemble Q-Learning in Large-Scale Wireless Networkscited by 4
  12. 2024Learning Uncertainty-Aware Temporally-Extended Actionscited by 3
18 more
  1. 2024Batch Ensemble for Variance Dependent Regret in Stochastic Banditscited by 3
  2. 2024Coverage Analysis of Multi-Environment Q-Learning Algorithms for Wireless Network Optimizationcited by 2
  3. 2021Balancing Value Underestimation and Overestimation with Realistic Actor-Criticcited by 1
  4. 2023Graph Exploration for Effective Multi-agent Q-Learningcited by 1
  5. 2024Oracle-Efficient Reinforcement Learning for Max Value Ensemblescited by 1
  6. 2024Broad Critic Deep Actor Reinforcement Learning for Continuous Controlcited by 1
  7. 2025CAMP in the Odyssey: Provably Robust Reinforcement Learning with Certified Radius Maximizationcited by 1
  8. 2025ADDQ: Adaptive Distributional Double Q-Learningcited by 1
  9. 2025Deep Double Q-learningcited by 1
  10. 2022Factors of Influence of the Overestimation Bias of Q-Learningcited by 0
  11. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  12. 2025An Arbitration Control for an Ensemble of Diversified DQN variants in Continual Reinforcement Learningcited by 0
  13. 2025UACER: An Uncertainty-Adaptive Critic Ensemble Framework for Robust Adversarial Reinforcement Learningcited by 0
  14. 2025Learning to Plan, Planning to Learn: Adaptive Hierarchical RL-MPC for Sample-Efficient Decision Makingcited by 0
  15. 2026SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfercited by 0
  16. 2026Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Errorcited by 0
  17. 2026Quantile of Means: A Bonus-Free Ensemble Method for Minimax Optimal Reinforcement Learningcited by 0
  18. 2026Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersectioncited by 0