mlsanityData to 7 Oct 2026

Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information

arXiv 2109.05198 ↗First posted 11 Sep 2021cs.LG math.OCcited by 16

Not on the map. Why?

Abstract (arXiv)

We present a novel adaptive optimization algorithm for large-scale machine learning problems. Equipped with a low-cost estimate of local curvature and Lipschitz smoothness, our method dynamically adapts the search direction and step-size. The search direction contains gradient information preconditioned by a well-scaled diagonal preconditioning matrix that captures the local curvature information. Our methodology does not require the tedious task of learning rate tuning, as the learning rate is updated automatically without adding an extra hyperparameter. We provide convergence guarantees on a comprehensive collection of optimization problems, including convex, strongly convex, and nonconvex problems, in both deterministic and stochastic regimes. We also conduct an extensive empirical evaluation on standard machine learning problems, justifying our algorithm's versatility and demonstrating its strong performance compared to other start-of-the-art first-order and second-order methods.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 29 of its 52 references that are in the map. First posted on arXiv 11 Sep 2021.

Builds on

29 of 52 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  6. 2019On the Convergence of Adam and Beyondcited by 834
  7. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  8. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  9. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  10. 2019SGD: General Analysis and Improved Ratescited by 206
  11. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  12. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
17 more
  1. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  2. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  3. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
  4. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 94
  5. 2019Adaptive Gradient Descent without Descentcited by 79
  6. 2017Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Informationcited by 41
  7. 2016Exact and Inexact Subsampled Newton Methods for Optimizationcited by 38
  8. 2017Second-Order Optimization for Non-Convex Machine Learning: An Empirical Studycited by 35
  9. 2016A Multi-Batch L-BFGS Method for Machine Learningcited by 30
  10. 2018New Convergence Aspects of Stochastic Gradient Algorithmscited by 30
  11. 2014Global Convergence of Online Limited Memory BFGScited by 18
  12. 2017An Investigation of Newton-Sketch and Subsampled Newton Methodscited by 18
  13. 2017A Newton-Based Method for Nonconvex Optimization with Fast Evasion of Saddle Pointscited by 12
  14. 2018Inexact Non-Convex Newton-Type Methodscited by 11
  15. 2018Efficient Distributed Hessian Free Algorithm for Large-scale Empirical Risk Minimization via Accumulating Sample Strategycited by 5
  16. 2019Scaling Up Quasi-Newton Algorithms: Communication Efficient Distributed SR1cited by 5
  17. 2020SONIA: A Symmetric Blockwise Truncated Optimization Algorithmcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 5; 2025: 2; 2026: 1.

16 papers.

Sort:

  1. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  2. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 30
  3. 2023Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtrackingcited by 9
  4. 2023Stochastic Gradient Descent with Preconditioned Polyak Step-sizecited by 7
  5. 2024HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimizationcited by 7
  6. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  7. 2022Stochastic Gradient Methods with Preconditioned Updatescited by 5
  8. 2023SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithmscited by 5
  9. 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
  10. 2022On Scaled Methods for Saddle Point Problemscited by 3
  11. 2025Bant: Byzantine Antidote via Trial Function and Trust Scorescited by 3
  12. 2023FOSI: Hybrid First and Second Order Optimizationcited by 1
4 more
  1. 2024Local Methods with Adaptivity via Scalingcited by 1
  2. 2024Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learningcited by 1
  3. 2026AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learningcited by 1
  4. 2024A second-order-like optimizer with adaptive gradient scaling for deep learningcited by 0