# Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent

arXiv: https://arxiv.org/abs/2002.10583 · HTML: https://mlsanity.com/paper/2002.10583/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2020.

## Built on by (most cited first, 24 of 24)

- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)
- [Improving Transformers with Probabilistic Attention Keys](https://mlsanity.com/paper/2110.08678/) (2021)
- [Resetting the Optimizer in Deep RL: An Empirical Study](https://mlsanity.com/paper/2306.17833/) (2023)
- [AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks](https://mlsanity.com/paper/2303.00565/) (2023)
- [Accelerated Gradient Descent Learning over Multiple Access Fading Channels](https://mlsanity.com/paper/2107.12452/) (2021)
- [How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies](https://mlsanity.com/paper/2110.07034/) (2021)
- [MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts](https://mlsanity.com/paper/2410.14574/) (2024)
- [Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization](https://mlsanity.com/paper/2110.09057/) (2021)
- [Last-iterate convergence analysis of stochastic momentum methods for neural networks](https://mlsanity.com/paper/2205.14811/) (2022)
- [Adaptive and Implicit Regularization for Matrix Completion](https://mlsanity.com/paper/2208.05640/) (2022)
- [PnP-CM: Consistency Models as Plug-and-Play Priors for Inverse Problems](https://mlsanity.com/paper/2509.22736/) (2025)
- [Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization](https://mlsanity.com/paper/2208.00579/) (2022)
- [An Adaptive Gradient Method with Energy and Momentum](https://mlsanity.com/paper/2203.12191/) (2022)
- [Expert Merging in Sparse Mixture of Experts with Nash Bargaining](https://mlsanity.com/paper/2510.16138/) (2025)
- [Accelerated Componentwise Gradient Boosting using Efficient Data Representation and Momentum-based Optimization](https://mlsanity.com/paper/2110.03513/) (2021)
- [Learning POD of Complex Dynamics Using Heavy-ball Neural ODEs](https://mlsanity.com/paper/2202.12373/) (2022)
- [Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimization](https://mlsanity.com/paper/2510.04988/) (2025)
- [EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization](https://mlsanity.com/paper/2605.25395/) (2026)
- [Stochastic Gradient Descent with Momentum is Algorithmically Stable](https://mlsanity.com/paper/2605.28517/) (2026)
- [Stability and Generalization of the Decentralized Stochastic Gradient Descent](https://mlsanity.com/paper/2102.01302/) (2021)
- [Convolutional Neural Network(CNN/ConvNet) in Stock Price Movement Prediction](https://mlsanity.com/paper/2106.01920/) (2021)
- [Momentum-inspired Low-Rank Coordinate Descent for Diagonally Constrained SDPs](https://mlsanity.com/paper/2106.08775/) (2021)
- [AIR-Net: Adaptive and Implicit Regularization Neural Network for Matrix Completion](https://mlsanity.com/paper/2110.07557/) (2021)
- [Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization](https://mlsanity.com/paper/2605.28585/) (2026)

## Abstract (arXiv)

Stochastic gradient descent (SGD) with constant momentum and its variants such as Adam are the optimization algorithms of choice for training deep neural networks (DNNs). Since DNN training is incredibly computationally expensive, there is great interest in speeding up the convergence. Nesterov accelerated gradient (NAG) improves the convergence rate of gradient descent (GD) for convex optimization using a specially designed momentum; however, it accumulates error when an inexact gradient is used (such as in SGD), slowing convergence at best and diverging at worst. In this paper, we propose Scheduled Restart SGD (SRSGD), a new NAG-style scheme for training DNNs. SRSGD replaces the constant momentum in SGD by the increasing momentum in NAG but stabilizes the iterations by resetting the momentum to zero according to a schedule. Using a variety of models and benchmarks for image classification, we demonstrate that, in training DNNs, SRSGD significantly improves convergence and generalization; for instance in training ResNet200 for ImageNet classification, SRSGD achieves an error rate of 20.93% vs. the benchmark of 22.13%. These improvements become more significant as the network grows deeper. Furthermore, on both CIFAR and ImageNet, SRSGD reaches similar or even better error rates with significantly fewer training epochs compared to the SGD baseline.