mlsanityData to 7 Oct 2026

Convergent and Efficient Deep Q Network Algorithm

arXiv 2106.15419 ↗First posted 29 Jun 2021cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Despite the empirical success of the deep Q network (DQN) reinforcement learning algorithm and its variants, DQN is still not well understood and it does not guarantee convergence. In this work, we show that DQN can indeed diverge and cease to operate in realistic settings. Although there exist gradient-based convergent methods, we show that they actually have inherent problems in learning dynamics which cause them to fail even in simple tasks. To overcome these problems, we propose a convergent DQN algorithm (C-DQN) that is guaranteed to converge and can work with large discount factors (0.9998). It learns robustly in difficult settings and can learn several difficult games in the Atari 2600 benchmark that DQN fails to solve. Our codes have been publicly released and can be used to reproduce our results.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 29 of its 58 references that are in the map. First posted on arXiv 29 Jun 2021.

Builds on

29 of 58 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  10. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  11. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  12. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
17 more
  1. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  2. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  3. 2020First return, then explorecited by 256
  4. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  5. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  6. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  7. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  8. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
  9. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  10. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  11. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  12. 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
  13. 2020Gradient Temporal-Difference Learning with Regularized Correctionscited by 38
  14. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  15. 2019Deep Residual Reinforcement Learningcited by 21
  16. 2017Convergent Tree Backup and Retrace with Function Approximationcited by 11
  17. 2019Deep Reinforcement Learning Control of Quantum Cartpolescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 0; 2025: 0; 2026: 1.

8 papers.

  1. 2023Controlled Decoding from Language Modelscited by 133
  2. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  3. 2023Reinforcement Unlearningcited by 16
  4. 2023Toward Efficient Gradient-Based Value Estimationcited by 4
  5. 2023Multi-Bellman operator for convergence of Q-learning with linear function approximationcited by 3
  6. 2023Emulation Learning for Neuromimetic Systemscited by 1
  7. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
  8. 2022Control of Continuous Quantum Systems with Many Degrees of Freedom based on Convergent Reinforcement Learningcited by 0