# On Convergence of Gradient Expected Sarsa(λ)

arXiv: https://arxiv.org/abs/2012.07199 · HTML: https://mlsanity.com/paper/2012.07199/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 4 of 4)

- [Constrained Update Projection Approach to Safe Policy Optimization](https://mlsanity.com/paper/2209.07089/) (2022)
- [CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learning](https://mlsanity.com/paper/2202.07565/) (2022)
- [Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features](https://mlsanity.com/paper/2409.12135/) (2024)
- [Finite Sample Analysis of Linear Temporal Difference Learning with Arbitrary Features](https://mlsanity.com/paper/2505.21391/) (2025)

## Abstract (arXiv)

We study the convergence of mathttExpected~Sarsa(λ) with linear function approximation. We show that applying the off-line estimate (multi-step bootstrapping) to mathttExpected~Sarsa(λ) is unstable for off-policy learning. Furthermore, based on convex-concave saddle-point framework, we propose a convergent mathttGradient~Expected~Sarsa(λ) (mathttGES(λ)) algorithm. The theoretical analysis shows that our mathttGES(λ) converges to the optimal solution at a linear convergence rate, which is comparable to extensive existing state-of-the-art gradient temporal difference learning algorithms. Furthermore, we develop a Lyapunov function technique to investigate how the step-size influences finite-time performance of mathttGES(λ), such technique of Lyapunov function can be potentially generalized to other GTD algorithms. Finally, we conduct experiments to verify the effectiveness of our mathttGES(λ).