# Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descent

arXiv: https://arxiv.org/abs/1607.01981 · HTML: https://mlsanity.com/paper/1607.01981/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2016.

## Built on by (most cited first, 26 of 26)

- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality](https://mlsanity.com/paper/2310.01035/) (2023)
- [Learning from Future: A Novel Self-Training Framework for Semantic Segmentation](https://mlsanity.com/paper/2209.06993/) (2022)
- [Correcting Momentum in Temporal Difference Learning](https://mlsanity.com/paper/2106.03955/) (2021)
- [Towards a universal mechanism for successful deep learning](https://mlsanity.com/paper/2309.07537/) (2023)
- [HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels](https://mlsanity.com/paper/2202.10716/) (2022)
- [Efficient shallow learning as an alternative to deep learning](https://mlsanity.com/paper/2211.11106/) (2022)
- [Meta-Learned Modality-Weighted Knowledge Distillation for Robust Multi-Modal Learning with Missing Data](https://mlsanity.com/paper/2405.07155/) (2024)
- [DATA: Multi-Disentanglement based Contrastive Learning for Open-World Semi-Supervised Deepfake Attribution](https://mlsanity.com/paper/2505.04384/) (2025)
- [The mechanism underlying successful deep learning](https://mlsanity.com/paper/2305.18078/) (2023)
- [Clothes-Changing Person Re-identification Based On Skeleton Dynamics](https://mlsanity.com/paper/2503.10759/) (2025)
- [Enhancing the accuracies by performing pooling decisions adjacent to the output layer](https://mlsanity.com/paper/2303.05800/) (2023)
- [Can we learn better with hard samples?](https://mlsanity.com/paper/2304.03486/) (2023)
- [Robust Semi-supervised Learning via f-Divergence and α-R'enyi Divergence](https://mlsanity.com/paper/2405.00454/) (2024)
- [Improving Adversarial Robustness for Free with Snapshot Ensemble](https://mlsanity.com/paper/2110.03124/) (2021)
- [UAV-based Crowd Surveillance in Post COVID-19 Era](https://mlsanity.com/paper/2111.14176/) (2021)
- [The Disharmony between BN and ReLU Causes Gradient Explosion, but is Offset by the Correlation between Activations](https://mlsanity.com/paper/2304.11692/) (2023)
- [Role of Delay in Brain Dynamics](https://mlsanity.com/paper/2410.11384/) (2024)
- [Error estimates between SGD with momentum and underdamped Langevin diffusion](https://mlsanity.com/paper/2410.17297/) (2024)
- [DNA: Dynamic Network Augmentation](https://mlsanity.com/paper/2112.09277/) (2021)
- [Two-stream Multi-dimensional Convolutional Network for Real-time Violence Detection](https://mlsanity.com/paper/2211.04255/) (2022)
- [A Scientific Machine Learning Approach for Predicting and Forecasting Battery Degradation in Electric Vehicles](https://mlsanity.com/paper/2410.14347/) (2024)
- [Enhanced NIRMAL Optimizer With Damped Nesterov Acceleration: A Comparative Analysis](https://mlsanity.com/paper/2508.16550/) (2025)
- [A Proportional-Integral Controller-Incorporated SGD Algorithm for High Efficient Latent Factor Analysis](https://mlsanity.com/paper/2508.17609/) (2025)
- [Riemannian Lyapunov Optimizer: A Unified Framework for Optimization](https://mlsanity.com/paper/2601.22284/) (2026)
- [Token-UNet: A New Case for Transformers Integration in Efficient and Interpretable 3D UNets for Brain Imaging Segmentation](https://mlsanity.com/paper/2602.20008/) (2026)

## Abstract (arXiv)

We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we term Regularised Gradient Descent, can converge more quickly than either Nesterov's algorithm or the classical momentum algorithm.