mlsanityData to 7 Oct 2026

Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning

arXiv 1611.01929 ↗First posted 7 Nov 2016cs.AI cs.LG stat.MLcited by 106

Not on the map. Why?

Abstract (arXiv)

Instability and variability of Deep Reinforcement Learning (DRL) algorithms tend to adversely affect their performance. Averaged-DQN is a simple extension to the DQN algorithm, based on averaging previously learned Q-values estimates, which leads to a more stable training procedure and improved performance by reducing approximation error variance in the target values. To understand the effect of the algorithm, we examine the source of value function estimation errors and provide an analytical comparison within a simplified model. We further present experiments on the Arcade Learning Environment benchmark that demonstrate significantly improved stability and performance due to the proposed extension.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 31; 2022: 15; 2023: 13; 2024: 21; 2025: 19; 2026: 5.

106 papers.

Sort:

By area:

  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  2. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  3. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  4. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  5. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  6. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  7. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
  8. 2024Towards Efficient Exact Optimization of Language Model Alignmentcited by 37
  9. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 36
  10. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  11. 2021Ensemble Bootstrapping for Q-Learningcited by 30
  12. 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 26
38 more
  1. 2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 22
  2. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  3. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 20
  4. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  5. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  6. 2021Regularized Softmax Deep Multi-Agent Q-Learningcited by 19
  7. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  8. 2022Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemblecited by 18
  9. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  10. 2023Ensemble Reinforcement Learning: A Surveycited by 16
  11. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  12. 2020Self-Imitation Learning via Generalized Lower Bound Q-learningcited by 13
  13. 2023Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agentscited by 12
  14. 2023Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedbackcited by 9
  15. 2022DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learningcited by 8
  16. 2022Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Taskscited by 8
  17. 2022Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue Systemcited by 8
  18. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  19. 2024A Survey Analyzing Generalization in Deep Reinforcement Learningcited by 8
  20. 2024SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learningcited by 8
  21. 2021Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilizationcited by 7
  22. 2021Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performancecited by 7
  23. 2024Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equationcited by 7
  24. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  25. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processescited by 6
  26. 2026EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Trainingcited by 6
  27. 2021A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasetscited by 5
  28. 2022Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworkscited by 5
  29. 2022Self-Activating Neural Ensembles for Continual Reinforcement Learningcited by 5
  30. 2023Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learningcited by 5
  31. 2023PASTA: Pretrained Action-State Transformer Agentscited by 5
  32. 2023Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Controlcited by 5
  33. 2024Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimizationcited by 5
  34. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  35. 2021Experience-Based Heuristic Search: Robust Motion Planning with Deep Q-Learningcited by 4
  36. 2021Maximum Entropy Reinforcement Learning with Mixture Policiescited by 4
  37. 2021On the Estimation Bias in Double Q-Learningcited by 4
  38. 2022Prediction Based Decision Making for Autonomous Highway Drivingcited by 4

50 most cited of 106 All 106 (JSON)