mlsanityData to 7 Oct 2026

Towards Characterizing Divergence in Deep Q-Learning

arXiv 1903.08894 ↗First posted 21 Mar 2019cs.LG cs.AIcited by 43

Not on the map. Why?

Abstract (arXiv)

Deep Q-Learning (DQL), a family of temporal difference algorithms for control, employs three techniques collectively known as the `deadly triad' in reinforcement learning: bootstrapping, off-policy learning, and function approximation. Prior work has demonstrated that together these can lead to divergence in Q-learning algorithms, but the conditions under which divergence occurs are not well-understood. In this note, we give a simple analysis based on a linear approximation to the Q-value updates, which we believe provides insight into divergence under the deadly triad. The central point in our analysis is to consider when the leading order approximation to the deep-Q update is or is not a contraction in the sup norm. Based on this analysis, we develop an algorithm which permits stable deep Q-learning for continuous control without any of the tricks conventionally used (such as target networks, adaptive gradient optimizers, or using multiple Q functions). We demonstrate that our algorithm performs above or near state-of-the-art on standard MuJoCo benchmarks from the OpenAI Gym.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 13; 2022: 8; 2023: 8; 2024: 6; 2025: 4; 2026: 2.

43 papers.

Sort:

By area:

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  3. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  4. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  5. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  6. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37
  7. 2021Regularized Behavior Value Estimationcited by 32
  8. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
  9. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  10. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  11. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  12. 2021HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanismcited by 19
31 more
  1. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 18
  2. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  3. 2021Functional Regularization for Reinforcement Learning via Learned Fourier Featurescited by 17
  4. 2023Behavior Alignment via Reward Function Optimizationcited by 16
  5. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  6. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  7. 2023Finite-Time Analysis of Whittle Index based Q-Learning for Restless Multi-Armed Bandits with Neural Network Function Approximationcited by 14
  8. 2021Correcting Momentum in Temporal Difference Learningcited by 12
  9. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
  10. 2021Convergent and Efficient Deep Q Network Algorithmcited by 8
  11. 2021Bridging the Gap Between Target Networks and Functional Regularizationcited by 7
  12. 2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7
  13. 2021Modularity in Reinforcement Learning via Algorithmic Independence in Credit Assignmentcited by 6
  14. 2023On the Performance of Temporal Difference Learning With Neural Networkscited by 6
  15. 2023Measuring and Mitigating Interference in Reinforcement Learningcited by 5
  16. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learningcited by 5
  17. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
  18. 2021On the Estimation Bias in Double Q-Learningcited by 4
  19. 2023Toward Efficient Gradient-Based Value Estimationcited by 4
  20. 2024Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learningcited by 4
  21. 2021Predictor-Corrector(PC) Temporal Difference(TD) Learning (PCTD)cited by 1
  22. 2022Deep Q-learning: a robust control approachcited by 1
  23. 2023The Ladder in Chaos: A Simple and Effective Improvement to General DRL Algorithms by Policy Path Trimming and Boostingcited by 1
  24. 2025Is Exploration or Optimization the Problem for Deep Reinforcement Learning?cited by 1
  25. 2026Less is More: Clustered Cross-Covariance Control for Offline RLcited by 1
  26. 2022Bridging the Gap Between Target Networks and Functional Regularizationcited by 0
  27. 2022Deep Reinforcement Learning for IRS Phase Shift Design in Spatiotemporally Correlated Environmentscited by 0
  28. 2022Control of Continuous Quantum Systems with Many Degrees of Freedom based on Convergent Reinforcement Learningcited by 0
  29. 2023Gauss-Newton Temporal Difference Learning with Nonlinear Function Approximationcited by 0
  30. 2024Probing Implicit Bias in Semi-gradient Q-learning: Visualizing the Effective Loss Landscapes via the Fokker--Planck Equationcited by 0
  31. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0