# Approximate Temporal Difference Learning is a Gradient Descent for Reversible Policies

arXiv: https://arxiv.org/abs/1805.00869 · HTML: https://mlsanity.com/paper/1805.00869/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2018.

## Built on by (most cited first, 10 of 10)

- [Distributed TD(0) with Almost No Communication](https://mlsanity.com/paper/2305.16246/) (2023)
- [Bootstrapped Representations in Reinforcement Learning](https://mlsanity.com/paper/2306.10171/) (2023)
- [On the Performance of Temporal Difference Learning With Neural Networks](https://mlsanity.com/paper/2312.05397/) (2023)
- [Towards a Better Understanding of Representation Dynamics under TD-learning](https://mlsanity.com/paper/2305.18491/) (2023)
- [Which Features are Best for Successor Features?](https://mlsanity.com/paper/2502.10790/) (2025)
- [A Robust widetildeO(1/√T) Rate for Unprojected TD Learning with Linear Function Approximation](https://mlsanity.com/paper/2506.01052/) (2025)
- [A Single Stepsize Suffices for Unprojected Linear TD(0): Simultaneous Robust and Fast Rates via Polyak--Ruppert Averaging](https://mlsanity.com/paper/2606.24981/) (2026)
- [Distributed TD(0) with Almost No Communication](https://mlsanity.com/paper/2104.07855/) (2021)
- [Bridging the Gap Between Average and Discounted TD Learning](https://mlsanity.com/paper/2605.02103/) (2026)
- [How I learned to stop worrying and love StopGrads: Stationarity, Convergence, and a case study on Flow Map Learning](https://mlsanity.com/paper/2609.16222/) (2026)

## Abstract (arXiv)

In reinforcement learning, temporal difference (TD) is the most direct algorithm to learn the value function of a policy. For large or infinite state spaces, exact representations of the value function are usually not available, and it must be approximated by a function in some parametric family. However, with nonlinear parametric approximations (such as neural networks), TD is not guaranteed to converge to a good approximation of the true value function within the family, and is known to diverge even in relatively simple cases. TD lacks an interpretation as a stochastic gradient descent of an error between the true and approximate value functions, which would provide such guarantees. We prove that approximate TD is a gradient descent provided the current policy is reversible. This holds even with nonlinear approximations. A policy with transition probabilities P(s,s') between states is reversible if there exists a function μ over states such that fracP(s,s')P(s',s)=fracμ(s')μ(s). In particular, every move can be undone with some probability. This condition is restrictive; it is satisfied, for instance, for a navigation problem in any unoriented graph. In this case, approximate TD is exactly a gradient descent of the Dirichlet norm, the norm of the difference of gradients between the true and approximate value functions. The Dirichlet norm also controls the bias of approximate policy gradient. These results hold even with no decay factor (γ=1) and do not rely on contractivity of the Bellman operator, thus proving stability of TD even with γ=1 for reversible policies.