On Convergence of Gradient Expected Sarsa(λ)

We study the convergence of mathttExpected~Sarsa(λ) with linear function approximation. We show that applying the off-line estimate (multi-step bootstrapping) to mathttExpected~Sarsa(λ) is unstable for off-policy learning. Furthermore, based on convex-concave saddle-point framework, we propose a convergent mathttGradient~Expected~Sarsa(λ) (mathttGES(λ)) algorithm. The theoretical analysis shows that our mathttGES(λ) converges to the optimal solution at a linear convergence rate, which is comparable to extensive existing state-of-the-art gradient temporal difference learning algorithms. Furthermore, we develop a Lyapunov function technique to investigate how the step-size influences finite-time performance of mathttGES(λ), such technique of Lyapunov function can be potentially generalized to other GTD algorithms. Finally, we conduct experiments to verify the effectiveness of our mathttGES(λ).