# Closing the gap between SVRG and TD-SVRG with Gradient Splitting

arXiv: https://arxiv.org/abs/2211.16237 · HTML: https://mlsanity.com/paper/2211.16237/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 14 of 14)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation](https://mlsanity.com/paper/1806.02450/) (2018)
- [Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learning](https://mlsanity.com/paper/1902.00923/) (2019)
- [Minimizing Finite Sums with the Stochastic Average Gradient](https://mlsanity.com/paper/1309.2388/) (2013)
- [Stochastic Variance Reduction Methods for Policy Evaluation](https://mlsanity.com/paper/1702.07944/) (2017)
- [Reanalysis of Variance Reduced Temporal Difference Learning](https://mlsanity.com/paper/2001.01898/) (2020)
- [On TD(0) with function approximation: Concentration bounds and a centered variant with exponential convergence](https://mlsanity.com/paper/1411.3224/) (2014)
- [Temporal Difference Learning as Gradient Splitting](https://mlsanity.com/paper/2010.14657/) (2020)
- [Stop Wasting My Gradients: Practical SVRG](https://mlsanity.com/paper/1511.01942/) (2015)
- [Variance-Reduced Off-Policy TDC Learning: Non-Asymptotic Convergence Analysis](https://mlsanity.com/paper/2010.13272/) (2020)
- [Convergent Tree Backup and Retrace with Function Approximation](https://mlsanity.com/paper/1705.09322/) (2017)
- [Optimal Lockdown for Pandemic Control](https://mlsanity.com/paper/2010.12923/) (2020)
- [SVRG for Policy Evaluation with Fewer Gradient Evaluations](https://mlsanity.com/paper/1906.03704/) (2019)

## Built on by (most cited first, 1 of 1)

- [Towards Parameter-Free Temporal Difference Learning](https://mlsanity.com/paper/2603.02577/) (2026)

## Abstract (arXiv)

Temporal difference (TD) learning is a policy evaluation in reinforcement learning whose performance can be enhanced by variance reduction methods. Recently, multiple works have sought to fuse TD learning with Stochastic Variance Reduced Gradient (SVRG) method to achieve a geometric rate of convergence. However, the resulting convergence rate is significantly weaker than what is achieved by SVRG in the setting of convex optimization. In this work we utilize a recent interpretation of TD-learning as the splitting of the gradient of an appropriately chosen function, thus simplifying the algorithm and fusing TD with SVRG. Our main result is a geometric convergence bound with predetermined learning rate of 1/8, which is identical to the convergence bound available for SVRG in the convex setting. Our theoretical findings are supported by a set of experiments.