# Asynchronous Stochastic Gradient Descent with Variance Reduction for Non-Convex Optimization

arXiv: https://arxiv.org/abs/1604.03584 · HTML: https://mlsanity.com/paper/1604.03584/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2016.

## Built on by (most cited first, 2 of 2)

- [SYNTHESIS: A Semi-Asynchronous Path-Integrated Stochastic Gradient Method for Distributed Learning in Computing Clusters](https://mlsanity.com/paper/2208.08425/) (2022)
- [Scaling up Stochastic Gradient Descent for Non-convex Optimisation](https://mlsanity.com/paper/2210.02882/) (2022)

## Abstract (arXiv)

We provide the first theoretical analysis on the convergence rate of the asynchronous stochastic variance reduced gradient (SVRG) descent algorithm on non-convex optimization. Recent studies have shown that the asynchronous stochastic gradient descent (SGD) based algorithms with variance reduction converge with a linear convergent rate on convex problems. However, there is no work to analyze asynchronous SGD with variance reduction technique on non-convex problem. In this paper, we study two asynchronous parallel implementations of SVRG: one is on a distributed memory system and the other is on a shared memory system. We provide the theoretical analysis that both algorithms can obtain a convergence rate of O(1/T), and linear speed up is achievable if the number of workers is upper bounded. V1,v2,v3 have been withdrawn due to reference issue, please refer the newest version v4.