mlsanityData to 14 Oct 2026

Robustness to Unbounded Smoothness of Generalized SignSGD

arXiv 2208.11195 ↗First posted 23 Aug 2022cs.LG math.OCcited by 81

Not on the map. Why?

Abstract (arXiv)

Traditional analyses in non-convex optimization typically rely on the smoothness assumption, namely requiring the gradients to be Lipschitz. However, recent evidence shows that this smoothness condition does not capture the properties of some deep learning objective functions, including the ones involving Recurrent Neural Networks and LSTMs. Instead, they satisfy a much more relaxed condition, with potentially unbounded smoothness. Under this relaxed assumption, it has been theoretically and empirically shown that the gradient-clipped SGD has an advantage over the vanilla one. In this paper, we show that clipping is not indispensable for Adam-type algorithms in tackling such scenarios: we theoretically prove that a generalized SignSGD algorithm can obtain similar convergence rates as SGD with clipping but does not need explicit clipping at all. This family of algorithms on one end recovers SignSGD and on the other end closely resembles the popular Adam algorithm. Our analysis underlines the critical role that momentum plays in analyzing SignSGD-type and Adam-type algorithms: it not only reduces the effects of noise, thus removing the need for large mini-batch in previous analyses of SignSGD-type algorithms, but it also substantially reduces the effects of unbounded smoothness and gradient norms. We also compare these algorithms with popular optimizers on a set of deep learning tasks, observing that we can match the performance of Adam while beating the others.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Builds on 39 of its 63 references that are in the map. First posted on arXiv 23 Aug 2022.

Builds on

39 of 63 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2021Convex optimizationcited by 2,515
  8. 2020Adaptive Federated Optimizationcited by 1,046
  9. 2019On the Convergence of Adam and Beyondcited by 841
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  11. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  12. 2014Deeply-Supervised Netscited by 367
27 more
  1. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  2. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  3. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  4. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  5. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  6. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  7. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
  8. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  9. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  10. 2020Momentum Improves Normalized SGDcited by 139
  11. 2017Non-convex Optimization for Machine Learningcited by 133
  12. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  13. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  14. 2018Tight Analyses for Non-Smooth Stochastic Gradient Descentcited by 97
  15. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  16. 2016Scale-Free Online Learningcited by 85
  17. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  18. 2015Beyond Convexity: Stochastic Quasi-Convex Optimizationcited by 76
  19. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  20. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  21. 2021Non-convex Distributionally Robust Optimization: Non-asymptotic Analysiscited by 39
  22. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
  23. 2015Scale-Free Algorithms for Online Linear Optimizationcited by 30
  24. 2018Efficient Decentralized Deep Learning by Dynamic Model Averagingcited by 30
  25. 2020LaProp: Separating Momentum and Adaptivity in Adamcited by 20
  26. 2018Universal Stagewise Learning for Non-Convex Problems with Convergence on Averaged Solutionscited by 9
  27. 2021Rethinking Adam: A Twofold Exponential Moving Average Approachcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 31; 2025: 17; 2026: 17.

81 papers.

Sort:

By area:

  1. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  2. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  3. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  4. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  5. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  6. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
  7. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  8. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  9. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 57
  10. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  11. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  12. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 49
38 more
  1. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  2. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  3. 2023Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGDcited by 41
  4. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  5. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  6. 2024Does SGD really happen in tiny subspaces?cited by 36
  7. 2023Parameter-Agnostic Optimization under Relaxed Smoothnesscited by 29
  8. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 26
  9. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 23
  10. 2023Variance-reduced Clipping for Non-convex Optimizationcited by 22
  11. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  12. 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
  13. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  14. 2024Towards Quantifying the Preconditioning Effect of Adamcited by 19
  15. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 19
  16. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  17. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  18. 2024Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimizationcited by 16
  19. 2024Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysiscited by 15
  20. 2024Revisiting Convergence of AdaGrad with Relaxed Assumptionscited by 15
  21. 2024On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyondcited by 14
  22. 2024Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variancecited by 14
  23. 2025Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under (L0, L1)-Smoothnesscited by 13
  24. 2024Gradient-Variation Online Learning under Generalized Smoothnesscited by 11
  25. 2024Understanding Adam Requires Better Rotation Dependent Assumptionscited by 11
  26. 2024An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothnesscited by 9
  27. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  28. 2023High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noisecited by 7
  29. 2024MGDA Converges under Generalized Smoothness, Provablycited by 7
  30. 2024Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-tailed Noise and Power of Symmetrycited by 7
  31. 2024A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothnesscited by 7
  32. 2025Unbiased Gradient Low-Rank Projectioncited by 7
  33. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7
  34. 2026MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-trainingcited by 6
  35. 2024R-SFLLM: Jamming Resilient Framework for Split Federated Learning with Large Language Modelscited by 5
  36. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  37. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 5
  38. 2023Rethinking SIGN Training: Provable Nonconvex Acceleration without First- and Second-Order Gradient Lipschitzcited by 4

50 most cited of 81 All 81 (JSON)