mlsanityData to 7 Oct 2026

Adaptive Gradient Methods with Local Guarantees

arXiv 2203.01400 ↗First posted 2 Mar 2022cs.LG math.OCcited by 8

Not on the map. Why?

Abstract (arXiv)

Adaptive gradient methods are the method of choice for optimization in machine learning and used to train the largest deep models. In this paper we study the problem of learning a local preconditioner, that can change as the data is changing along the optimization trajectory. We propose an adaptive gradient method that has provable adaptive regret guarantees vs. the best local preconditioner. To derive this guarantee, we prove a new adaptive regret bound in online learning that improves upon previous adaptive online learning methods. We demonstrate the robustness of our method in automatically choosing the optimal learning rate schedule for popular benchmarking tasks in vision and language domains. Without the need to manually tune a learning rate schedule, our method can, in a single run, achieve comparable and stable task accuracy as a fine-tuned optimizer.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 35 of its 60 references that are in the map. First posted on arXiv 2 Mar 2022.

Builds on

35 of 60 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  5. 2019On the Convergence of Adam and Beyondcited by 834
  6. 2019Introduction to Online Convex Optimizationcited by 793
  7. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  8. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  9. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  10. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  11. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  12. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
23 more
  1. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  2. 2018Adaptive Online Learning in Dynamic Environmentscited by 130
  3. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  4. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  5. 2015Strongly Adaptive Online Learningcited by 102
  6. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  7. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  8. 2014Input Warping for Bayesian Optimization of Non-stationary Functionscited by 66
  9. 2016Improved Strongly Adaptive Online Learning using Coin Bettingcited by 48
  10. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 39
  11. 2020Minimizing Dynamic Regret and Adaptive Regret Simultaneouslycited by 30
  12. 2017Hyperparameter Optimization: A Spectral Approachcited by 24
  13. 2018WNGrad: Learn the Learning Rate in Gradient Descentcited by 24
  14. 2019Adaptive Regret of Convex and Smooth Functionscited by 24
  15. 2020Learning compositional functions via multiplicative weight updatescited by 20
  16. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20
  17. 2021Adversarial Tracking Control via Strongly Adaptive Online Learning with Memorycited by 14
  18. 2021Acceleration via Fractal Learning Rate Schedulescited by 13
  19. 2022Online Control of Unknown Time-Varying Dynamical Systemscited by 13
  20. 2016DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networkscited by 9
  21. 2019Lecture Notes: Optimization for Machine Learningcited by 8
  22. 2016Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogatescited by 7
  23. 2019Extreme Tensoring for Low-Memory Preconditioningcited by 5

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 2; 2025: 2.

8 papers.

  1. 2023Optimal Stochastic Non-smooth Non-convex Optimization through Online-to-Non-convex Conversioncited by 42
  2. 2023Mechanic: A Learning Rate Tunercited by 20
  3. 2022Projection-free Adaptive Regret with Membership Oraclescited by 13
  4. 2022On the Computational Efficiency of Adaptive and Dynamic Regret Minimizationcited by 2
  5. 2024Lifelong Event Detection with Embedding Space Separation and Compactioncited by 2
  6. 2024Adaptive Regret for Bandits Made Possible: Two Queries Sufficecited by 1
  7. 2025Non-stationary Bandit Convex Optimization: A Comprehensive Studycited by 1
  8. 2025Accuracy-Constrained CNN Pruning for Efficient and Reliable EEG-Based Seizure Detectioncited by 0