mlsanityData to 14 Oct 2026

Deep Double Q-learning

arXiv 2507.00275 ↗First posted 30 Jun 2025cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Double Q-learning is a classical control algorithm that mitigates the maximization bias of Q-learning. To do so, it explicitly trains two independent action-value functions and uses them to decouple action-selection and action-evaluation when computing bootstrap targets. Double DQN adapts target bootstrap decoupling to deep reinforcement learning (RL), but explicitly trains only a single action-value function and does not fully decouple its estimators. Consequently, the two estimators remain correlated, and overestimation persists. In this paper, we introduce Deep Double Q-learning (DDQL), a deep RL algorithm that explicitly trains two Q-functions through Double Q-learning. DDQL stabilizes training through a combination of techniques, including lower replay ratios, longer target network update intervals, and shared layers. Across 57 Atari 2600 games, DDQL improves aggregate performance over Double DQN, outperforming it on 47 games while further reducing overestimation. In addition, we study key design choices when adapting Double Q-learning to deep RL, including the network architecture, replay ratio, and minibatch sampling strategies.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 47 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

27 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL6 Mar

Other references · 26

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  9. 2016Deep Exploration via Bootstrapped DQNcited by 527
  10. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  11. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  12. 2020Revisiting Fundamentals of Experience Replaycited by 175
14 more
  1. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  2. 2019When to use parametric models in reinforcement learning?cited by 118
  3. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  4. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  5. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  6. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 57
  7. 2019ChainerRL: A Deep Reinforcement Learning Librarycited by 51
  8. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  9. 2021Ensemble Bootstrapping for Q-Learningcited by 31
  10. 2022The Phenomenon of Policy Churncited by 28
  11. 2020Self-correcting Q-Learningcited by 9
  12. 2020Decorrelated Double Q-learningcited by 3
  13. 2022Addressing Maximization Bias in Reinforcement Learning with Two-Sample Testingcited by 3
  14. 2020Cross Learning in Deep Q-Networkscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Accelerating Q-learning through Efficient Value-Sharing across Actionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning
  2. 2024Finite-Time Analysis of Simultaneous Double Q-learning
  3. 2024Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks
  4. 2025ADDQ: Adaptive Distributional Double Q-Learning
  5. 2026A Controlled Study of Double DQN and Dueling DQN Under Cross-Environment Transfer
  6. 2024Time-Scale Separation in Q-Learning: Extending TD(triangle) for Action-Value Function Decomposition
  7. 2024VQC-Based Reinforcement Learning with Data Re-uploading: Performance and Trainability
  8. 2021Reinforcement Learning In Two Player Zero Sum Simultaneous Action Games