mlsanityData to 14 Oct 2026

TD Convergence: An Optimization Perspective

arXiv 2306.17750 ↗First posted 30 Jun 2023cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We study the convergence behavior of the celebrated temporal-difference (TD) learning algorithm. By looking at the algorithm through the lens of optimization, we first argue that TD can be viewed as an iterative optimization algorithm where the function to be minimized changes per iteration. By carefully investigating the divergence displayed by TD on a classical counter example, we identify two forces that determine the convergent or divergent behavior of the algorithm. We next formalize our discovery in the linear TD setting with quadratic loss and prove that convergence of TD hinges on the interplay between these two forces. We extend this optimization perspective to prove convergence of TD in a much broader setting than just linear approximation and squared loss. Our results provide a theoretical explanation for the successful application of TD in reinforcement learning.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 23 of its 60 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

23 of 60 references.

References · 23

  1. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  2. 2016Input Convex Neural Networkscited by 414
  3. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  4. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  5. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  6. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  7. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  8. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  9. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 57
  10. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  11. 2020Gradient Temporal-Difference Learning with Regularized Correctionscited by 38
  12. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
11 more
  1. 2020Logistic Q-Learningcited by 34
  2. 2020Finite-Sample Analysis of Proximal Gradient TD Algorithmscited by 32
  3. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  4. 2019Target-Based Temporal Difference Learningcited by 20
  5. 2020Finite-Sample Analysis of Stochastic Approximation Using Smooth Convex Envelopescited by 19
  6. 2022Regularized Q-learningcited by 16
  7. 2019Zap Q-Learning With Nonlinear Function Approximationcited by 13
  8. 2020Regularized Off-Policy TD-Learningcited by 10
  9. 2021Faster Deep Reinforcement Learning with Slower Online Networkcited by 7
  10. 2017A First Empirical Study of Emphatic Temporal Difference Learningcited by 6
  11. 2019A Convergent Off-Policy Temporal Difference Algorithmcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCEcited by 14
  2. 2024Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximationcited by 10
  3. 2025Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iterationcited by 6
  4. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  5. 2024Learning the Target Network in Function Spacecited by 1
  6. 2025Addressing Finite-Horizon MDPs via Low-Rank Tensor Value Approximationcited by 1
  7. 2026Periodic Regularized Q-Learningcited by 1
  8. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0
  9. 2025Convergence of TD(0) under Polynomial Mixing with Nonlinear Function Approximationcited by 0
  10. 2025Risk-Averse Total-Reward Reinforcement Learningcited by 0
  11. 2025Unrolling Dynamic Programming via Graph Filterscited by 0
  12. 2025Imitate Optimal Policy: Prevail and Induce Action Collapse in Policy Gradientcited by 0
4 more
  1. 2026Gradient Iterated Temporal-Difference Learningcited by 0
  2. 2026Bayesian learning for the stochastic shortest path problemcited by 0
  3. 2026A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learningcited by 0
  4. 2026On the Convergence of Semi-Gradient TD: A Mean Field Perspectivecited by 0