# Schedule Based Temporal Difference Algorithms

arXiv: https://arxiv.org/abs/2111.11768 · HTML: https://mlsanity.com/paper/2111.11768/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 27 references that are in the map.
- First posted on arXiv 23 Nov 2021.

## Builds on (most cited first, 5 of 5)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Stability of Stochastic Approximations with `Controlled Markov' Noise and Temporal Difference Learning](https://mlsanity.com/paper/1504.06043/) (2015)
- [Online Off-policy Prediction](https://mlsanity.com/paper/1811.02597/) (2018)
- [Two Timescale Stochastic Approximation with Controlled Markov noise and Off-policy temporal difference learning](https://mlsanity.com/paper/1503.09105/) (2015)
- [An Incremental Off-policy Search in a Model-free Markov Decision Process Using a Single Sample Path](https://mlsanity.com/paper/1801.10287/) (2018)

## Abstract (arXiv)

Learning the value function of a given policy from data samples is an important problem in Reinforcement Learning. TD(λ) is a popular class of algorithms to solve this problem. However, the weights assigned to different n-step returns in TD(λ), controlled by the parameter λ, decrease exponentially with increasing n. In this paper, we present a λ-schedule procedure that generalizes the TD(λ) algorithm to the case when the parameter λ could vary with time-step. This allows flexibility in weight assignment, i.e., the user can specify the weights assigned to different n-step returns by choosing a sequence λtt ≥ 1. Based on this procedure, we propose an on-policy algorithm - TD(λ)-schedule, and two off-policy algorithms - GTD(λ)-schedule and TDC(λ)-schedule, respectively. We provide proofs of almost sure convergence for all three algorithms under a general Markov noise framework.