mlsanityData to 7 Oct 2026

Accelerate Distributed Stochastic Descent for Nonconvex Optimization with Momentum

arXiv 2110.00625 ↗First posted 1 Oct 2021cs.LG cs.AI math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Momentum method has been used extensively in optimizers for deep learning. Recent studies show that distributed training through K-step averaging has many nice properties. We propose a momentum method for such model averaging approaches. At each individual learner level traditional stochastic gradient is applied. At the meta-level (global learner level), one momentum term is applied and we call it block momentum. We analyze the convergence and scaling properties of such momentum methods. Our experimental results show that block momentum not only accelerates training, but also achieves better results.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 22 references that are in the map. First posted on arXiv 1 Oct 2021.

Builds on

13 of 22 references.

References · 13

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Going Deeper with Convolutionscited by 4,231
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  4. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  5. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
  6. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
  7. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 91
  8. 2013Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programmingcited by 74
  9. 2016Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimizationcited by 34
  10. 2016A Lyapunov Analysis of Momentum Methods in Optimizationcited by 28
  11. 2015Asynchronous Parallel Stochastic Gradient Descent - A Numeric Core for Scalable Distributed Machine Learning Algorithmscited by 4
  12. 2018A Diffusion Approximation Theory of Momentum SGD in Nonconvex Optimizationcited by 2
1 more
  1. 2018Towards Understanding Acceleration Tradeoff between Momentum and Asynchrony in Nonconvex Stochastic Optimizationcited by 1

Built on by

Not cited yet.