mlsanityData to 14 Oct 2026

Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimization

arXiv 2410.14054 ↗First posted 17 Oct 2024math.OC stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent studies have shown that many nonconvex machine learning problems satisfy a generalized-smooth condition that extends beyond traditional smooth nonconvex optimization. However, the existing algorithms are not fully adapted to such generalized-smooth nonconvex geometry and encounter significant technical limitations on their convergence analysis. In this work, we first analyze the convergence of adaptively normalized gradient descent under function geometries characterized by generalized-smoothness and generalized PL condition, revealing the advantage of adaptive gradient normalization. Our results provide theoretical insights into adaptive normalization across various scenarios.For stochastic generalized-smooth nonconvex optimization, we propose Independent-Adaptively Normalized Stochastic Gradient Descent algorithm, which leverages adaptive gradient normalization, independent sampling, and gradient clipping to achieve an O(ε-4) sample complexity under relaxed noise assumptions. Experiments on large-scale nonconvex generalized-smooth problems demonstrate the fast convergence of our algorithm.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

40 of 56 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  6. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  7. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  8. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  9. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  10. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  11. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  12. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
28 more
  1. 2020Momentum Improves Normalized SGDcited by 139
  2. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  3. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  4. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  5. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 81
  6. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 57
  7. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  8. 2023Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGDcited by 41
  9. 2021Non-convex Distributionally Robust Optimization: Non-asymptotic Analysiscited by 39
  10. 2016Calculus of the exponent of Kurdyka-Lojasiewicz inequality and its applications to linear convergence of first-order methodscited by 35
  11. 2023Generalized-Smooth Nonconvex Optimization is As Efficient As Smooth Nonconvex Optimizationcited by 35
  12. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30
  13. 2023Parameter-Agnostic Optimization under Relaxed Smoothnesscited by 29
  14. 2024Methods for Convex (L0,L1)-Smooth Optimization: Clipping, Acceleration, and Adaptivitycited by 25
  15. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  16. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  17. 2024Directional Smoothness and Gradient Methods: Convergence and Adaptivitycited by 17
  18. 2024Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysiscited by 15
  19. 2024Revisiting Convergence of AdaGrad with Relaxed Assumptionscited by 15
  20. 2024On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyondcited by 14
  21. 2024Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variancecited by 14
  22. 2024Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reductioncited by 13
  23. 2024Gradient-Variation Online Learning under Generalized Smoothnesscited by 11
  24. 2024An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothnesscited by 9
  25. 2023Trust Region Methods For Nonconvex Stochastic Optimization Beyond Lipschitz Smoothnesscited by 8
  26. 2023Rethinking SIGN Training: Provable Nonconvex Acceleration without First- and Second-Order Gradient Lipschitzcited by 4
  27. 2024A New First-Order Meta-Learning Algorithm with Convergence Guaranteescited by 4
  28. 2024Generalized Smooth Stochastic Variational Inequalities: Almost Sure Convergence and Convergence Ratescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026On the Role of Batch Size in Stochastic Conditional Gradient Methodscited by 7
  2. 2025On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)-Smoothness: An SDE Approachcited by 4
  3. 2026Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learningcited by 0
  4. 2026Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothnesscited by 0