# Investigating Alternatives to the Root Mean Square for Adaptive Gradient Methods

arXiv: https://arxiv.org/abs/2106.05449 · HTML: https://mlsanity.com/paper/2106.05449/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Recurrent Neural Network Regularization](https://mlsanity.com/paper/1409.2329/) (2014)
- [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://mlsanity.com/paper/1902.09843/) (2019)
- [Exploring the Limits of Language Modeling](https://mlsanity.com/paper/1602.02410/) (2016)

## Abstract (arXiv)

Adam is an adaptive gradient method that has experienced widespread adoption due to its fast and reliable training performance. Recent approaches have not offered significant improvement over Adam, often because they do not innovate upon one of its core features: normalization by the root mean square (RMS) of recent gradients. However, as noted by Kingma and Ba (2015), any number of Lp normalizations are possible, with the RMS corresponding to the specific case of p=2. In our work, we theoretically and empirically characterize the influence of different Lp norms on adaptive gradient methods for the first time. We show mathematically how the choice of p influences the size of the steps taken, while leaving other desirable properties unaffected. We evaluate Adam with various Lp norms on a suite of deep learning benchmarks, and find that p > 2 consistently leads to improved learning speed and final performance. The choices of p=3 or p=6 also match or outperform state-of-the-art methods in all of our experiments.