mlsanityData to 7 Oct 2026

AdaLoss: A computationally-efficient and provably convergent adaptive gradient method

arXiv 2109.08282 ↗First posted 17 Sep 2021stat.ML cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We propose a computationally-friendly adaptive learning rate schedule, "AdaLoss", which directly uses the information of the loss function to adjust the stepsize in gradient descent methods. We prove that this schedule enjoys linear convergence in linear regression. Moreover, we provide a linear convergence guarantee over the non-convex regime, in the context of two-layer over-parameterized neural networks. If the width of the first-hidden layer in the two-layer networks is sufficiently large (polynomially), then AdaLoss converges robustly to the global minimum in polynomial time. We numerically verify the theoretical results and extend the scope of the numerical experiments by considering applications in LSTM models for text clarification and policy gradients for control problems.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 17 Sep 2021.

Builds on

26 of 39 references.

References · 26

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2014Convex Optimization: Algorithms and Complexitycited by 569
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  9. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  10. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  11. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  12. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
14 more
  1. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  2. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  3. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 124
  4. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  5. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
  6. 2018Online Adaptive Methods, Universality and Accelerationcited by 51
  7. 2017Variants of RMSProp and Adagrad with Logarithmic Regret Boundscited by 50
  8. 2017Online to Offline Conversions, Universality and Adaptive Minibatch Sizescited by 39
  9. 2018Discrete Adversarial Attacks and Submodular Optimization with Applications to Text Classificationcited by 34
  10. 2016Barzilai-Borwein Step Size for Stochastic Gradient Descentcited by 33
  11. 2015Scale-Free Algorithms for Online Linear Optimizationcited by 30
  12. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  13. 2019Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Networkcited by 27
  14. 2018WNGrad: Learn the Learning Rate in Gradient Descentcited by 24

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 1.

3 papers.

  1. 2023Robust Implicit Regularization via Weight Normalizationcited by 10
  2. 2021Provable Regret Bounds for Deep Online Learning and Controlcited by 6
  3. 2022Adaptively Weighted Data Augmentation Consistency Regularization for Robust Optimization under Concept Shiftcited by 3