mlsanityData to 7 Oct 2026

Breaking the Deadly Triad with a Target Network

arXiv 2101.08862 ↗First posted 21 Jan 2021cs.LGcited by 47

Not on the map. Why?

Abstract (arXiv)

The deadly triad refers to the instability of a reinforcement learning algorithm when it employs off-policy learning, function approximation, and bootstrapping simultaneously. In this paper, we investigate the target network as a tool for breaking the deadly triad, providing theoretical support for the conventional wisdom that a target network stabilizes training. We first propose and analyze a novel target network update rule which augments the commonly used Polyak-averaging style update with two projections. We then apply the target network and ridge regularization in several divergent algorithms and show their convergence to regularized TD fixed points. Those algorithms are off-policy with linear function approximation and bootstrapping, spanning both policy evaluation and control, as well as both discounted and average-reward settings. In particular, we provide the first convergent linear Q-learning algorithms under nonrestrictive and changing behavior policies without bi-level optimization.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 24 of its 65 references that are in the map. First posted on arXiv 21 Jan 2021.

Builds on

24 of 65 references.

References · 24

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  5. 2019Challenges of Real-World Reinforcement Learningcited by 282
  6. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  7. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  8. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  9. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 120
  10. 2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
  11. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  12. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
12 more
  1. 2019Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learningcited by 67
  2. 2017Stochastic Variance Reduction Methods for Policy Evaluationcited by 56
  3. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  4. 2019A Finite-Time Analysis of Q-Learning with Neural Network Function Approximationcited by 47
  5. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  6. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  7. 2020Finite-sample Analysis of Greedy-GQ with Linear Function Approximation under Markovian Noisecited by 17
  8. 2019A Unified Switching System Perspective and O.D.E. Analysis of Q-Learning Algorithmscited by 14
  9. 2017On Convergence of some Gradient-based Temporal-Differences Algorithms for Off-Policy Learningcited by 11
  10. 2019Zap Q-Learning With Nonlinear Function Approximationcited by 10
  11. 2020Regularized Off-Policy TD-Learningcited by 10
  12. 2019A Convergent Off-Policy Temporal Difference Algorithmcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 7; 2023: 7; 2024: 10; 2025: 10; 2026: 6.

47 papers.

Sort:

By area:

  1. 2024Simplifying Deep Temporal Difference Learningcited by 71
  2. 2022Target Network and Truncation Overcome The Deadly Triad in Q-Learningcited by 31
  3. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  4. 2021Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatchcited by 17
  5. 2022Regularized Q-learningcited by 16
  6. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  7. 2022Rethinking ValueDice: Does It Really Improve Performance?cited by 15
  8. 2022On the Convergence of SARSA with Linear Function Approximationcited by 15
  9. 2023TD Convergence: An Optimization Perspectivecited by 15
  10. 2021Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximationcited by 13
  11. 2023Why Target Networks Stabilise Temporal Difference Methodscited by 13
  12. 2024Offline Imitation Learning Through Graph Search and Retrievalcited by 12
35 more
  1. 2023Bootstrapped Representations in Reinforcement Learningcited by 11
  2. 2025Linear Q-Learning Does Not Diverge in L2: Convergence Rates to a Bounded Setcited by 11
  3. 2021Truncated Emphatic Temporal Difference Methods for Prediction and Controlcited by 10
  4. 2024Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximationcited by 10
  5. 2021Convergent and Efficient Deep Q Network Algorithmcited by 8
  6. 2021Bridging the Gap Between Target Networks and Functional Regularizationcited by 7
  7. 2021Faster Deep Reinforcement Learning with Slower Online Networkcited by 7
  8. 2023Revisiting a Design Choice in Gradient Temporal Difference Learningcited by 7
  9. 2022Consolidated Adaptive T-soft Update for Deep Reinforcement Learningcited by 5
  10. 2024RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learningcited by 5
  11. 2024Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Featurescited by 5
  12. 2025Finite Sample Analysis of Linear Temporal Difference Learning with Arbitrary Featurescited by 5
  13. 2025Extensions of Robbins-Siegmund Theorem with Applications in Reinforcement Learningcited by 5
  14. 2022A Note on Target Q-learning For Solving Finite MDPs with A Generative Oraclecited by 4
  15. 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
  16. 2023Multi-Bellman operator for convergence of Q-learning with linear function approximationcited by 3
  17. 2023A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using L-λ Smoothnesscited by 2
  18. 2025Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learningcited by 2
  19. 2026Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processescited by 2
  20. 2026A Switching System Theory of Q-Learning with Linear Function Approximationcited by 2
  21. 2026Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamicscited by 2
  22. 2021The Impact of Data Distribution on Q-learning with Function Approximationcited by 1
  23. 2024Regularized Q-Learning with Linear Function Approximationcited by 1
  24. 2024Learning the Target Network in Function Spacecited by 1
  25. 2024Deep deterministic policy gradient with symmetric data augmentation for lateral attitude tracking control of a fixed-wing aircraftcited by 1
  26. 2025Is Exploration or Optimization the Problem for Deep Reinforcement Learning?cited by 1
  27. 2022Bridging the Gap Between Target Networks and Functional Regularizationcited by 0
  28. 2024Debiased Model-based Interactive Recommendationcited by 0
  29. 2025Goal-oriented Transmission Scheduling: Structure-guided DRL with a Unified Dual On-policy and Off-policy Approachcited by 0
  30. 2025Analysis of Off-Policy n-Step TD-Learning with Linear Function Approximationcited by 0
  31. 2025Deep Learning Agents Trained For Avoidance Behave Like Hawks And Dovescited by 0
  32. 2025Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Explorationcited by 0
  33. 2026Periodic Regularized Q-Learningcited by 0
  34. 2026Target-Aligned Reinforcement Learningcited by 0
  35. 2026Behavior-Induced Mirror-Prox Temporal-Difference Learning for Faster Off-Policy Predictioncited by 0