# Stop Wasting My Gradients: Practical SVRG

arXiv: https://arxiv.org/abs/1511.01942 · HTML: https://mlsanity.com/paper/1511.01942/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2015.

## Built on by (most cited first, 14 of 14)

- [A Generalizable Approach to Learning Optimizers](https://mlsanity.com/paper/2106.00958/) (2021)
- [AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimation](https://mlsanity.com/paper/2204.13170/) (2022)
- [Variance Reduced Training with Stratified Sampling for Forecasting Models](https://mlsanity.com/paper/2103.02062/) (2021)
- [SVRG Meets AdaGrad: Painless Variance Reduction](https://mlsanity.com/paper/2102.09645/) (2021)
- [A Variance Controlled Stochastic Method with Biased Estimation for Faster Non-convex Optimization](https://mlsanity.com/paper/2102.09893/) (2021)
- [Stochastic Steffensen method](https://mlsanity.com/paper/2211.15310/) (2022)
- [An Effective Dynamic Gradient Calibration Method for Continual Learning](https://mlsanity.com/paper/2407.20956/) (2024)
- [Closing the gap between SVRG and TD-SVRG with Gradient Splitting](https://mlsanity.com/paper/2211.16237/) (2022)
- [Stochastic Ratios Tracking Algorithm for Large Scale Machine Learning Problems](https://mlsanity.com/paper/2305.09978/) (2023)
- [Acoustic Model Optimization over Multiple Data Sources: Merging and Valuation](https://mlsanity.com/paper/2410.15620/) (2024)
- [Switch and Conquer: Efficient Algorithms By Switching Stochastic Gradient Oracles For Decentralized Saddle Point Problems](https://mlsanity.com/paper/2309.00997/) (2023)
- [A Theoretical and Empirical Study on the Convergence of Adam with an "Exact" Constant Step Size in Non-Convex Settings](https://mlsanity.com/paper/2309.08339/) (2023)
- [SOREL: A Stochastic Algorithm for Spectral Risks Minimization](https://mlsanity.com/paper/2407.14618/) (2024)
- [SVRG and Beyond via Posterior Correction](https://mlsanity.com/paper/2512.01930/) (2025)

## Abstract (arXiv)

We present and analyze several strategies for improving the performance of stochastic variance-reduced gradient (SVRG) methods. We first show that the convergence rate of these methods can be preserved under a decreasing sequence of errors in the control variate, and use this to derive variants of SVRG that use growing-batch strategies to reduce the number of gradient calculations required in the early iterations. We further (i) show how to exploit support vectors to reduce the number of gradient computations in the later iterations, (ii) prove that the commonly-used regularized SVRG iteration is justified and improves the convergence rate, (iii) consider alternate mini-batch selection strategies, and (iv) consider the generalization error of the method.