# k-SVRG: Variance Reduction for Large Scale Optimization

arXiv: https://arxiv.org/abs/1805.00982 · HTML: https://mlsanity.com/paper/1805.00982/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2018.

## Built on by (most cited first, 1 of 1)

- [Better Methods and Theory for Federated Learning: Compression, Client Selection and Heterogeneity](https://mlsanity.com/paper/2207.00392/) (2022)

## Abstract (arXiv)

Variance reduced stochastic gradient (SGD) methods converge significantly faster than the vanilla SGD counterpart. However, these methods are not very practical on large scale problems, as they either i) require frequent passes over the full data to recompute gradients---without making any progress during this time (like for SVRG), or ii)~they require additional memory that can surpass the size of the input problem (like for SAGA). In this work, we propose k-SVRG that addresses these issues by making best use of the available memory and minimizes the stalling phases without progress. We prove linear convergence of k-SVRG on strongly convex problems and convergence to stationary points on non-convex problems. Numerical experiments show the effectiveness of our method.