# New Versions of Gradient Temporal Difference Learning

arXiv: https://arxiv.org/abs/2109.04033 · HTML: https://mlsanity.com/paper/2109.04033/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization](https://mlsanity.com/paper/1806.00877/) (2018)
- [Gradient Temporal-Difference Learning with Regularized Corrections](https://mlsanity.com/paper/2007.00611/) (2020)
- [A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning](https://mlsanity.com/paper/2104.13844/) (2021)
- [Distributed Policy Evaluation Under Multiple Behavior Strategies](https://mlsanity.com/paper/1312.7606/) (2013)
- [Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning](https://mlsanity.com/paper/1612.02516/) (2016)
- [On the Exponential Stability of Primal-Dual Gradient Dynamics](https://mlsanity.com/paper/1803.01825/) (2018)
- [Fast Multi-Agent Temporal-Difference Learning via Homotopy Stochastic Primal-Dual Optimization](https://mlsanity.com/paper/1908.02805/) (2019)
- [Primal-Dual Distributed Temporal Difference Learning](https://mlsanity.com/paper/1805.07918/) (2018)

## Built on by (most cited first, 6 of 6)

- [Regularized Q-learning](https://mlsanity.com/paper/2202.05404/) (2022)
- [A primal-dual perspective for distributed TD-learning](https://mlsanity.com/paper/2310.00638/) (2023)
- [Analysis of Off-Policy Multi-Step TD-Learning with Linear Function Approximation](https://mlsanity.com/paper/2402.15781/) (2024)
- [Control Theoretic Analysis of Temporal Difference Learning](https://mlsanity.com/paper/2112.14417/) (2021)
- [Analysis of Off-Policy n-Step TD-Learning with Linear Function Approximation](https://mlsanity.com/paper/2502.08941/) (2025)
- [R-GTD: A Geometric Analysis of Gradient Temporal-Difference Learning in Singular Regimes](https://mlsanity.com/paper/2601.20599/) (2026)

## Abstract (arXiv)

Sutton, Szepesv'ari and Maei introduced the first gradient temporal-difference (GTD) learning algorithms compatible with both linear function approximation and off-policy training. The goal of this paper is (a) to propose some variants of GTDs with extensive comparative analysis and (b) to establish new theoretical analysis frameworks for the GTDs. These variants are based on convex-concave saddle-point interpretations of GTDs, which effectively unify all the GTDs into a single framework, and provide simple stability analysis based on recent results on primal-dual gradient dynamics. Finally, numerical comparative analysis is given to evaluate these approaches.