# Guided parallelized stochastic gradient descent for delay compensation

arXiv: https://arxiv.org/abs/2101.07259 · HTML: https://mlsanity.com/paper/2101.07259/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 17 Jan 2021.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Convex Optimization: Algorithms and Complexity](https://mlsanity.com/paper/1405.4980/) (2014)
- [Sparsified SGD with Memory](https://mlsanity.com/paper/1809.07599/) (2018)
- [Asynchronous Stochastic Gradient Descent with Delay Compensation](https://mlsanity.com/paper/1609.08326/) (2016)
- [Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptions](https://mlsanity.com/paper/1902.00908/) (2019)
- [GIANT: Globally Improved Approximate Newton Method for Distributed Optimization](https://mlsanity.com/paper/1709.03528/) (2017)
- [DINGO: Distributed Newton-Type Method for Gradient-Norm Optimization](https://mlsanity.com/paper/1901.05134/) (2019)
- [GraphLab: A New Framework For Parallel Machine Learning](https://mlsanity.com/paper/1408.2041/) (2014)
- [The Convergence of Stochastic Gradient Descent in Asynchronous Shared Memory](https://mlsanity.com/paper/1803.08841/) (2018)

## Built on by (most cited first, 3 of 3)

- [SMOTified-GAN for class imbalanced pattern classification problems](https://mlsanity.com/paper/2108.03235/) (2021)
- [Data Distribution-based Curriculum Learning](https://mlsanity.com/paper/2402.07352/) (2024)
- [SGD method for entropy error function with smoothing l0 regularization for neural networks](https://mlsanity.com/paper/2405.18552/) (2024)

## Abstract (arXiv)

Stochastic gradient descent (SGD) algorithm and its variations have been effectively used to optimize neural network models. However, with the rapid growth of big data and deep learning, SGD is no longer the most suitable choice due to its natural behavior of sequential optimization of the error function. This has led to the development of parallel SGD algorithms, such as asynchronous SGD (ASGD) and synchronous SGD (SSGD) to train deep neural networks. However, it introduces a high variance due to the delay in parameter (weight) update. We address this delay in our proposed algorithm and try to minimize its impact. We employed guided SGD (gSGD) that encourages consistent examples to steer the convergence by compensating the unpredictable deviation caused by the delay. Its convergence rate is also similar to A/SSGD, however, some additional (parallel) processing is required to compensate for the delay. The experimental results demonstrate that our proposed approach has been able to mitigate the impact of delay for the quality of classification accuracy. The guided approach with SSGD clearly outperforms sequential SGD and even achieves the accuracy close to sequential SGD for some benchmark datasets.