# Implicit Temporal Differences

arXiv: https://arxiv.org/abs/1412.6734 · HTML: https://mlsanity.com/paper/1412.6734/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2014.

## Built on by (most cited first, 1 of 1)

- [Implicit Updates for Average-Reward Temporal Difference Learning](https://mlsanity.com/paper/2510.06149/) (2025)

## Abstract (arXiv)

In reinforcement learning, the TD(λ) algorithm is a fundamental policy evaluation method with an efficient online implementation that is suitable for large-scale problems. One practical drawback of TD(λ) is its sensitivity to the choice of the step-size. It is an empirically well-known fact that a large step-size leads to fast convergence, at the cost of higher variance and risk of instability. In this work, we introduce the implicit TD(λ) algorithm which has the same function and computational cost as TD(λ), but is significantly more stable. We provide a theoretical explanation of this stability and an empirical evaluation of implicit TD(λ) on typical benchmark tasks. Our results show that implicit TD(λ) outperforms standard TD(λ) and a state-of-the-art method that automatically tunes the step-size, and thus shows promise for wide applicability.