A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range
Not on the map. Why?
Abstract (arXiv)
We make contributions towards improving adaptive-optimizer performance. Our improvements are based on suppression of the range of adaptive stepsizes in the AdaBelief optimizer. Firstly, we show that the particular placement of the parameter epsilon within the update expressions of AdaBelief reduces the range of the adaptive stepsizes, making AdaBelief closer to SGD with momentum. Secondly, we extend AdaBelief by further suppressing the range of the adaptive stepsizes. To achieve the above goal, we perform mutual layerwise vector projections between the gradient g_t and its first momentum m_t before using them to estimate the second momentum. The new optimization method is referred to as Aida. Thirdly, extensive experimental results show that Aida outperforms nine optimizers when training transformers and LSTMs for NLP, and VGG and ResNet for image classification over CIAF10 and CIFAR100 while matching the best performance of the nine methods when training WGAN-GP models for image generation tasks. Furthermore, Aida produces higher validation accuracies than AdaBelief for training ResNet18 over ImageNet. Code is available <a href="https://github.com/guoqiang-x-zhang/AidaOptimizer">at this URL</a>
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 16 of its 29 references that are in the map. First posted on arXiv 24 Mar 2022.
Builds on
16 of 29 references.
References · 16
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2017Improved Training of Wasserstein GANscited by 2,223
- 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
- 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
- 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
- 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
- 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
4 more
- 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
- 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
- 2020On the distance between two neural networks and the stability of learningcited by 46
- 2019Why are Adaptive Methods Good for Attention Models?cited by 35
Built on by
Citing papers per year: 2025: 1.
1 paper.