mlsanityData to 14 Oct 2026

On the Distributional Properties of Adaptive Gradients

arXiv 2105.07222 ↗First posted 15 May 2021cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Adaptive gradient methods have achieved remarkable success in training deep neural networks on a wide variety of tasks. However, not much is known about the mathematical and statistical properties of this family of methods. This work aims at providing a series of theoretical analyses of its statistical properties justified by experiments. In particular, we show that when the underlying gradient obeys a normal distribution, the variance of the magnitude of the textitupdate is an increasing and bounded function of time and does not diverge. This work suggests that the divergence of variance is not the cause of the need for warm up of the Adam optimizer, contrary to what is believed in the current literature.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 15 May 2021.

Builds on

31 of 43 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  11. 2017Decoupled Weight Decay Regularizationcited by 1,831
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
19 more
  1. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  2. 2020Designing Network Design Spacescited by 960
  3. 2019On the Convergence of Adam and Beyondcited by 841
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  5. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  6. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 263
  7. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  8. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  9. 2020The Heavy-Tail Phenomenon in SGDcited by 124
  10. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
  11. 2018Training Tips for the Transformer Modelcited by 85
  12. 2019Optimization for deep learning: theory and algorithmscited by 70
  13. 2018Convergence and Dynamical Behavior of the ADAM Algorithm for Non-Convex Stochastic Optimizationcited by 49
  14. 2021Strength of Minibatch Noise in SGDcited by 39
  15. 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 37
  16. 2019On the adequacy of untuned warmup for adaptive optimizationcited by 35
  17. 2018A general system of differential equations to model first order adaptive algorithmscited by 28
  18. 2016An empirical analysis of the optimization of deep network loss surfacescited by 21
  19. 2018Theory of Deep Learning IIb: Optimization Properties of SGDcited by 12

Built on by

Not cited yet.