mlsanityData to 7 Oct 2026

Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks

arXiv 2209.07670 ↗First posted 16 Sep 2022cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

In temporal-difference reinforcement learning algorithms, variance in value estimation can cause instability and overestimation of the maximal target value. Many algorithms have been proposed to reduce overestimation, including several recent ensemble methods, however none have shown success in sample-efficient learning through addressing estimation variance as the root cause of overestimation. In this paper, we propose MeanQ, a simple ensemble method that estimates target values as ensemble means. Despite its simplicity, MeanQ shows remarkable sample efficiency in experiments on the Atari Learning Environment benchmark. Importantly, we find that an ensemble of size 5 sufficiently reduces estimation variance to obviate the lagging target network, eliminating it as a source of bias and further gaining sample efficiency. We justify intuitively and empirically the design choices in MeanQ, including the necessity of independent experience sampling. On a set of 26 benchmark Atari environments, MeanQ outperforms all tested baselines, including the best available baseline, SUNRISE, at 100K interaction steps in 16/26 environments, and by 68% on average. MeanQ also outperforms Rainbow DQN at 500K steps in 21/26 environments, and by 49% on average, and achieves average human-level performance using 200K (±100K) interaction steps. Our implementation is available at https://github.com/indylab/MeanQ.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 32 of its 65 references that are in the map. First posted on arXiv 16 Sep 2022.

Builds on

32 of 65 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
20 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  3. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  4. 2016Deep Exploration via Bootstrapped DQNcited by 518
  5. 2019Model-Based Reinforcement Learning for Ataricited by 495
  6. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  7. 2017Noisy Networks for Explorationcited by 262
  8. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  9. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  10. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  11. 2019When to use parametric models in reinforcement learning?cited by 118
  12. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  13. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  14. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  15. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  16. 2021Ensemble Bootstrapping for Q-Learningcited by 30
  17. 2021Understanding the World Through Actioncited by 17
  18. 2015Emphatic Temporal-Difference Learningcited by 16
  19. 2018Revisiting the Softmax Bellman Operator: New Benefits and New Perspectivecited by 2
  20. 2021Temporal-Difference Value Estimation via Uncertainty-Guided Soft Updatescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 6; 2025: 2; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  2. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  3. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  4. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processescited by 6
  5. 2024The Curse of Diversity in Ensemble-Based Explorationcited by 6
  6. 2023Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learningcited by 5
  7. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learningcited by 4
  8. 2024Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learningcited by 1
  9. 2025Effective Reinforcement Learning for Reasoning in Language Modelscited by 1
  10. 2025RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Modelscited by 1
  11. 2024Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learningcited by 0
  12. 2024Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RLcited by 0
1 more
  1. 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learningcited by 0