mlsanityData to 14 Oct 2026

Last-Iterate Convergence Rate of Normalized Gradient Descent under H"older Smoothness

arXiv 2610.06070 ↗First posted 5 Oct 2026math.OC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Normalized gradient descent is a widely studied adaptive optimization method. Most existing analyses focus on the best iterate or a weighted average of the iterates, whereas practical implementations typically return the last iterate. In this paper, we study the last-iterate convergence of normalized gradient descent for convex, (nu,M_nu)-H"older-smooth objectives. For a constant stepsize, we establish an upper bound of Obigl((log2(T)/T)(1+nu)/2bigr), which contains a logarithmic overhead relative to the known Obigl(T-(1+nu)/2bigr) guarantees for the best and weighted-average iterates. For nu = 0, this overhead is known to be unavoidable. We complement this analysis with numerical results based on the performance estimation problem (PEP), investigating the finite-horizon worst-case behavior in the smooth setting and whether the logarithmic overhead reflects an intrinsic limitation of constant-step normalized gradient descent. We then show that a linearly decreasing stepsize yields a last-iterate guarantee of Obigl(T-(1+nu)/2bigr), matching the order of the best-iterate/weighted-average guarantees without requiring knowledge of nu and M_nu.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 34 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

26 of 34 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2025Muon is Scalable for LLM Trainingcited by 428
  7. 2024The Road Less Scheduledcited by 163
  8. 2020Momentum Improves Normalized SGDcited by 139
  9. 2018Tight Analyses for Non-Smooth Stochastic Gradient Descentcited by 97
  10. 2017Lower Bounds for Finding Stationary Points II: First-Order Methodscited by 58
  11. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 43
  12. 2023Revisiting the Last-Iterate Convergence of Stochastic Gradient Methodscited by 38
14 more
  1. 2015Smooth Strongly Convex Interpolation and Exact Worst-case Performance of First-order Methodscited by 37
  2. 2025The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Trainingcited by 33
  3. 2023Exact convergence rate of the last iterate in subgradient methodscited by 25
  4. 2023Normalized Gradients for Allcited by 20
  5. 2017Convergence Rates for Deterministic and Stochastic Subgradient Methods Without Lipschitz Continuitycited by 19
  6. 2021High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noisecited by 19
  7. 2024Optimizing (L0, L1)-Smooth Functions by Gradient Methodscited by 19
  8. 2024Directional Smoothness and Gradient Methods: Convergence and Adaptivitycited by 17
  9. 2022PEPit: computer-assisted worst-case analyses of first-order optimization methods in Pythoncited by 11
  10. 2024On the Last-Iterate Convergence of Shuffling Gradient Methodscited by 11
  11. 2021On the Last Iterate Convergence of Momentum Methodscited by 10
  12. 2024Exact Convergence rate of the subgradient method by using Polyak step sizecited by 4
  13. 2026Gradient Descent's Last Iterate is Often (slightly) Suboptimalcited by 4
  14. 2026Last Iterate Convergence of AdaGrad-Norm for Convex Non-Smooth Optimizationcited by 3

Built on by

Not cited yet.