mlsanityData to 14 Oct 2026

Deep Reinforcement Learning with Double Q-learning

arXiv 1509.06461 ↗First posted 22 Sep 2015cs.LGcited by 1,745

Not on the map. Why?

Cited by 472 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the recent DQN algorithm, which combines Q-learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q-learning algorithm, which was introduced in a tabular setting, can be generalized to work with large-scale function approximation. We propose a specific adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games.

Key facts. Cited by 1,745 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Sep 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 3; 2019: 7; 2020: 7; 2021: 342; 2022: 313; 2023: 302; 2024: 285; 2025: 288; 2026: 198.

1,745 papers.

Sort:

By area:

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  2. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  3. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  4. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  6. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  7. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  9. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  10. 2021Mastering Atari Games with Limited Datacited by 237
  11. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  12. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
38 more
  1. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  2. 2022Online Decision Transformercited by 196
  3. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 194
  4. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  5. 2021Reinforcement Learning with Prototypical Representationscited by 179
  6. 2023Understanding plasticity in neural networkscited by 174
  7. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  8. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  9. 2021Offline RL Without Off-Policy Evaluationcited by 151
  10. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  11. 2021The Benchmark Lotterycited by 136
  12. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  13. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 122
  14. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 111
  15. 2021Recent Advances in Reinforcement Learning in Financecited by 110
  16. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  17. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  18. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  19. 2021Deep Generative Models in Engineering Design: A Reviewcited by 105
  20. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  21. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 94
  22. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  23. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  24. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  25. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  26. 2021Conservative Offline Distributional Reinforcement Learningcited by 79
  27. 2024NetLLM: Adapting Large Language Models for Networkingcited by 78
  28. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
  29. 2021Off-Belief Learningcited by 75
  30. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  31. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 73
  32. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  33. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 72
  34. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  35. 2023Heterogeneous-Agent Reinforcement Learningcited by 69
  36. 2021DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learningcited by 66
  37. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  38. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65

50 most cited of 1,745 All 1,745 (JSON)