# AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights

arXiv: https://arxiv.org/abs/2006.08217 · HTML: https://mlsanity.com/paper/2006.08217/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 10 of 10)

- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021)
- [Uncertainty Calibration for Ensemble-Based Debiasing Methods](https://mlsanity.com/paper/2111.04104/) (2021)
- [AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs](https://mlsanity.com/paper/2210.06364/) (2022)
- [AdaInject: Injection Based Adaptive Gradient Descent Optimizers for Convolutional Neural Networks](https://mlsanity.com/paper/2109.12504/) (2021)
- [Conditional De-Identification of 3D Magnetic Resonance Images](https://mlsanity.com/paper/2110.09927/) (2021)
- [Many-MobileNet: Multi-Model Augmentation for Robust Retinal Disease Classification](https://mlsanity.com/paper/2412.02825/) (2024)
- [AmCLR: Unified Augmented Learning for Cross-Modal Representations](https://mlsanity.com/paper/2412.07979/) (2024)
- [OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers](https://mlsanity.com/paper/2607.04033/) (2026)

## Abstract (arXiv)

Normalization techniques are a boon for modern deep learning. They let weights converge more quickly with often better generalization performances. It has been argued that the normalization-induced scale invariance among the weights provides an advantageous ground for gradient descent (GD) optimizers: the effective step sizes are automatically reduced over time, stabilizing the overall training procedure. It is often overlooked, however, that the additional introduction of momentum in GD optimizers results in a far more rapid reduction in effective step sizes for scale-invariant weights, a phenomenon that has not yet been studied and may have caused unwanted side effects in the current practice. This is a crucial issue because arguably the vast majority of modern deep neural networks consist of (1) momentum-based GD (e.g. SGD or Adam) and (2) scale-invariant parameters. In this paper, we verify that the widely-adopted combination of the two ingredients lead to the premature decay of effective step sizes and sub-optimal model performances. We propose a simple and effective remedy, SGDP and AdamP: get rid of the radial component, or the norm-increasing direction, at each optimizer step. Because of the scale invariance, this modification only alters the effective step sizes without changing the effective update directions, thus enjoying the original convergence properties of GD optimizers. Given the ubiquity of momentum GD and scale invariance in machine learning, we have evaluated our methods against the baselines on 13 benchmarks. They range from vision tasks like classification (e.g. ImageNet), retrieval (e.g. CUB and SOP), and detection (e.g. COCO) to language modelling (e.g. WikiText) and audio classification (e.g. DCASE) tasks. We verify that our solution brings about uniform gains in those benchmarks. Source code is available at https://github.com/clovaai/AdamP.