Precise Convergence Speed of Clipped SGD
Not on the map. Why?
Abstract (arXiv)
We present a tightened convergence analysis of clipped gradient descent on (L0, L1)-smooth functions, with quantitative constants. Building on the ideas of Koloskova et al (2023), we refactor several case disjunctions to reveal the central role of a control of the bias derived from fundamental properties of ℓ2-projection, simplifying proofs. We also extend the domain of validity from η ≤ 1 / (9 β) to η < 1 /β where β = L0 + c L1 for clipping constant c, which matches the more traditional analysis of smooth functions. We strengthen the convergence criterion from left( mint < T mathbbE[lVert nabla f(xt) rVert2] right) to left( frac1T sumt < T mathbbE[lVert nabla f(xt) rVert2] right) with matching speed, and lower the final achievable loss from O(min(σ2/c, σ)) to the more precise 6 min(σ2 /c, 3 σ).
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 6 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
5 of 6 references.
References · 5
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
- 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
- 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
- 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
- 2026Avoiding Bias in Clipped SGD for Overparameterized Models under Generalized Smoothnesscited by 2
Built on by
Not cited yet.