mlsanityData to 7 Oct 2026

Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothness

arXiv 2506.01913 ↗First posted 2 Jun 2025cs.LG stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of (L0,L1)-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal O(n-1/4) convergence rate by leveraging a momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning, which we dub Clipped Scion, and demonstrate their properties on image classification and language modeling. The code is available at https://github.com/LIONS-EPFL/ClippedScion.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

32 of 51 references.

References · 32

  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2017Large Batch Training of Convolutional Networkscited by 529
  3. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  4. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  5. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  6. 2024Old Optimizer, New Norm: An Anthologycited by 173
  7. 2020Momentum Improves Normalized SGDcited by 136
  8. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  9. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 88
  10. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  11. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  12. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
20 more
  1. 2025A Note on the Convergence of Muoncited by 78
  2. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  3. 2024Scalable Optimization in the Modular Normcited by 64
  4. 2025Dion: Distributed Orthonormalized Updatescited by 64
  5. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 55
  6. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  7. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 50
  8. 2018Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximizationcited by 47
  9. 2023Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGDcited by 41
  10. 2020The Geometry of Sign Gradient Descentcited by 39
  11. 2023Parameter-Agnostic Optimization under Relaxed Smoothnesscited by 29
  12. 2020On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspectivecited by 25
  13. 2022Orthogonalising gradients to speed up neural network optimisationcited by 25
  14. 2024Methods for Convex (L0,L1)-Smooth Optimization: Clipping, Acceleration, and Adaptivitycited by 25
  15. 2013An Almost-Linear-Time Algorithm for Approximate Max Flow in Undirected Graphs, and its Multicommodity Generalizationscited by 18
  16. 2023Normalized Gradients for Allcited by 18
  17. 2024Optimizing (L0, L1)-Smooth Functions by Gradient Methodscited by 18
  18. 2025Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under (L0, L1)-Smoothnesscited by 12
  19. 2023The Frank-Wolfe algorithm: a short introductioncited by 9
  20. 2017The duality structure gradient descent algorithm: analysis and applications to neural networkscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 11.

17 papers.

Sort:

  1. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 23
  2. 2025LiMuon: Light and Fast Muon Optimizer for Large Modelscited by 20
  3. 2026Improved Convergence Rates of Muon Optimizer for Nonconvex Optimizationcited by 9
  4. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  5. 2026On the Role of Batch Size in Stochastic Conditional Gradient Methodscited by 6
  6. 2025Optimal Scaling Needs Optimal Normcited by 5
  7. 2025Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimizationcited by 4
  8. 2026Muon Converges under Heavy-Tailed Noise: Nonconvex H"older-Smooth Empirical Risk Minimizationcited by 2
  9. 2026Optimistic Dual Averaging Unifies Modern Optimizerscited by 2
  10. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  11. 2026Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Methodcited by 2
  12. 2026MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Modelscited by 2
5 more
  1. 2026Distance-Aware Muon: Adaptive Step Scaling for Normalized Optimizationcited by 1
  2. 2026Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Successcited by 1
  3. 2025Adaptive Conditional Gradient Descentcited by 0
  4. 2026Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectivescited by 0
  5. 2026Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?cited by 0