mlsanityData to 14 Oct 2026

Deep Reinforcement Learning and the Deadly Triad

arXiv 1812.02648 ↗First posted 6 Dec 2018cs.AI cs.LGcited by 174

Not on the map. Why?

Cited by 54 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We know from reinforcement learning theory that temporal difference learning can fail in certain cases. Sutton and Barto (2018) identify a deadly triad of function approximation, bootstrapping, and off-policy learning. When these three properties are combined, learning can diverge with the value estimates becoming unbounded. However, several algorithms successfully combine these three properties, which indicates that there is at least a partial gap in our understanding. In this work, we investigate the impact of the deadly triad in practice, in the context of a family of popular deep reinforcement learning models - deep Q-networks trained with experience replay - analysing how the components of this system play a role in the emergence of the deadly triad, and in the agent's performance

Key facts. Cited by 174 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 4; 2021: 27; 2022: 32; 2023: 28; 2024: 19; 2025: 34; 2026: 29.

174 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  4. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  5. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  6. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  7. 2022Bayesian Structure Learning with Generative Flow Networkscited by 153
  8. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 132
  9. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 79
  10. 2024Simplifying Deep Temporal Difference Learningcited by 78
  11. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
  12. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
38 more
  1. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
  2. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  3. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  4. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 59
  5. 2023Empirical Design in Reinforcement Learningcited by 55
  6. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  7. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  8. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  9. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 45
  10. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  11. 2021Training Larger Networks for Deep Reinforcement Learningcited by 38
  12. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 33
  13. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  14. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 32
  15. 2025Self-Improving Embodied Foundation Modelscited by 32
  16. 2021Ensemble Bootstrapping for Q-Learningcited by 31
  17. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  18. 2022deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networkscited by 28
  19. 2021Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Surveycited by 24
  20. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  21. 2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 23
  22. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  23. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  24. 2025GEM: A Gym for Agentic LLMscited by 23
  25. 2020Average-reward model-free reinforcement learning: a systematic review and literature mappingcited by 22
  26. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  27. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  28. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  29. 2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
  30. 2021Safe Continuous Control with Constrained Model-Based Policy Optimizationcited by 17
  31. 2021Functional Regularization for Reinforcement Learning via Learned Fourier Featurescited by 17
  32. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 17
  33. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimizationcited by 17
  34. 2022Critic Sequential Monte Carlocited by 16
  35. 2022Adversarial Cheap Talkcited by 16
  36. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  37. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 15
  38. 2022Uncovering Instabilities in Variational-Quantum Deep Q-Networkscited by 14

50 most cited of 174 All 174 (JSON)