# Asynchronous Parallel Stochastic Gradient Descent - A Numeric Core for Scalable Distributed Machine Learning Algorithms

arXiv: https://arxiv.org/abs/1505.04956 · HTML: https://mlsanity.com/paper/1505.04956/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2015.

## Built on by (most cited first, 4 of 4)

- [An Invitation to Distributed Quantum Neural Networks](https://mlsanity.com/paper/2211.07056/) (2022)
- [Convolutional Neural Nets in Chemical Engineering: Foundations, Computations, and Applications](https://mlsanity.com/paper/2101.04869/) (2021)
- [Performance and Energy Consumption of Parallel Machine Learning Algorithms](https://mlsanity.com/paper/2305.00798/) (2023)
- [Accelerate Distributed Stochastic Descent for Nonconvex Optimization with Momentum](https://mlsanity.com/paper/2110.00625/) (2021)

## Abstract (arXiv)

The implementation of a vast majority of machine learning (ML) algorithms boils down to solving a numerical optimization problem. In this context, Stochastic Gradient Descent (SGD) methods have long proven to provide good results, both in terms of convergence and accuracy. Recently, several parallelization approaches have been proposed in order to scale SGD to solve very large ML problems. At their core, most of these approaches are following a map-reduce scheme. This paper presents a novel parallel updating algorithm for SGD, which utilizes the asynchronous single-sided communication paradigm. Compared to existing methods, Asynchronous Parallel Stochastic Gradient Descent (ASGD) provides faster (or at least equal) convergence, close to linear scaling and stable accuracy.