# Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum

arXiv: https://arxiv.org/abs/2012.02188 · HTML: https://mlsanity.com/paper/2012.02188/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2020.

## Built on by (most cited first, 8 of 8)

- [How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studies](https://mlsanity.com/paper/2110.07034/) (2021)
- [Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization](https://mlsanity.com/paper/2110.09057/) (2021)
- [Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs](https://mlsanity.com/paper/2204.08621/) (2022)
- [Adaptive and Implicit Regularization for Matrix Completion](https://mlsanity.com/paper/2208.05640/) (2022)
- [Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/2309.15244/) (2023)
- [Learning POD of Complex Dynamics Using Heavy-ball Neural ODEs](https://mlsanity.com/paper/2202.12373/) (2022)
- [Practical and Fast Momentum-Based Power Methods](https://mlsanity.com/paper/2108.09264/) (2021)
- [AIR-Net: Adaptive and Implicit Regularization Neural Network for Matrix Completion](https://mlsanity.com/paper/2110.07557/) (2021)

## Abstract (arXiv)

Momentum plays a crucial role in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well-calibrated constant. However, tuning hyperparameters for momentum can be a significant computational burden. In this paper, we propose a novel adaptive momentum for improving DNNs training; this adaptive momentum, with no momentum related hyperparameter required, is motivated by the nonlinear conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this new adaptive momentum eliminates the need for the momentum hyperparameter calibration, allows a significantly larger learning rate, accelerates DNN training, and improves final accuracy and robustness of the trained DNNs. For instance, SGD with this adaptive momentum reduces classification errors for training ResNet110 for CIFAR10 and CIFAR100 from 5.25% to 4.64% and 23.75% to 20.03%, respectively. Furthermore, SGD with the new adaptive momentum also benefits adversarial training and improves adversarial robustness of the trained DNNs.