# GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training

arXiv: https://arxiv.org/abs/1312.6186 · HTML: https://mlsanity.com/paper/1312.6186/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2013.

## Built on by (most cited first, 7 of 7)

- [Anarchic Federated Learning](https://mlsanity.com/paper/2108.09875/) (2021)
- [AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregation](https://mlsanity.com/paper/2205.13797/) (2022)
- [Distributed stochastic optimization with large delays](https://mlsanity.com/paper/2107.02919/) (2021)
- [Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks](https://mlsanity.com/paper/2112.05387/) (2021)
- [Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo](https://mlsanity.com/paper/2606.22768/) (2026)
- [Scaling up Stochastic Gradient Descent for Non-convex Optimisation](https://mlsanity.com/paper/2210.02882/) (2022)
- [Substituting Proof of Work in Blockchain with Training-Verified Collaborative Model Computation](https://mlsanity.com/paper/2508.12138/) (2025)

## Abstract (arXiv)

The ability to train large-scale neural networks has resulted in state-of-the-art performance in many areas of computer vision. These results have largely come from computational break throughs of two forms: model parallelism, e.g. GPU accelerated training, which has seen quick adoption in computer vision circles, and data parallelism, e.g. A-SGD, whose large scale has been used mostly in industry. We report early experiments with a system that makes use of both model parallelism and data parallelism, we call GPU A-SGD. We show using GPU A-SGD it is possible to speed up training of large convolutional neural networks useful for computer vision. We believe GPU A-SGD will make it possible to train larger networks on larger training sets in a reasonable amount of time.