mlsanityData to 7 Oct 2026

Training Deep Learning Models with Norm-Constrained LMOs

arXiv 2502.07529 ↗First posted 11 Feb 2025cs.LG math.OCcited by 177

Not on the map. Why?

Abstract (arXiv)

In this work, we study optimization methods that leverage the linear minimization oracle (LMO) over a norm-ball. We propose a new stochastic family of algorithms that uses the LMO to adapt to the geometry of the problem and, perhaps surprisingly, show that they can be applied to unconstrained problems. The resulting update rule unifies several existing optimization methods under a single framework. Furthermore, we propose an explicit choice of norm for deep architectures, which, as a side benefit, leads to the transferability of hyperparameters across model sizes. Experimentally, we demonstrate significant speedups on nanoGPT training using our algorithm, Scion, without any reliance on Adam. The proposed method is memory-efficient, requiring only one set of model weights and one set of gradients, which can be stored in half-precision. The code is available at https://github.com/LIONS-EPFL/scion .

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Builds on 36 of its 66 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

36 of 66 references.

References · 36

  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  6. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  9. 2017Large Batch Training of Convolutional Networkscited by 529
  10. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  11. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 303
  12. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
24 more
  1. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  2. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  3. 2024Old Optimizer, New Norm: An Anthologycited by 173
  4. 2020Momentum Improves Normalized SGDcited by 136
  5. 2023A Spectral Condition for Feature Learningcited by 101
  6. 2015Beyond Convexity: Stochastic Quasi-Convex Optimizationcited by 75
  7. 2024Modular Duality in Deep Learningcited by 71
  8. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  9. 2024Scalable Optimization in the Modular Normcited by 64
  10. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  11. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  12. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 50
  13. 2015Preconditioned Stochastic Gradient Descentcited by 48
  14. 2018Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximizationcited by 47
  15. 2023Benchmarking Neural Network Training Algorithmscited by 46
  16. 2020The Geometry of Sign Gradient Descentcited by 39
  17. 2023Exact convergence rate of the last iterate in subgradient methodscited by 24
  18. 2020Deep Neural Network Training with Frank-Wolfecited by 22
  19. 2017A Unified Approach to Adaptive Regularization in Online and Stochastic Optimizationcited by 19
  20. 2013An Almost-Linear-Time Algorithm for Approximate Max Flow in Undirected Graphs, and its Multicommodity Generalizationscited by 18
  21. 2023Normalized Gradients for Allcited by 18
  22. 2024Curvature-Informed SGD via General Purpose Lie-Group Preconditionerscited by 16
  23. 2020Stochastic Normalized Gradient Descent with Momentum for Large-Batch Trainingcited by 9
  24. 2017The duality structure gradient descent algorithm: analysis and applications to neural networkscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 56; 2026: 121.

177 papers.

Sort:

By area:

  1. 2025Muon is Scalable for LLM Trainingcited by 381
  2. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  3. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  4. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  5. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 66
  6. 2025Dion: Distributed Orthonormalized Updatescited by 64
  7. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  8. 2025In Search of Adam's Secret Saucecited by 40
  9. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  10. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
  11. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 34
  12. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 30
38 more
  1. 2025On the Convergence of Muon and Beyondcited by 28
  2. 2026Controlled LLM Training on Spectral Spherecited by 28
  3. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 28
  4. 2025An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variantscited by 25
  5. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 24
  6. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 23
  7. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 22
  8. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  9. 2025LiMuon: Light and Fast Muon Optimizer for Large Modelscited by 20
  10. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  11. 2025Training Transformers with Enforced Lipschitz Constantscited by 18
  12. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  13. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 17
  14. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 16
  15. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 16
  16. 2025FedMuon: Accelerating Federated Learning with Matrix Orthogonalizationcited by 15
  17. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 15
  18. 2026The Newton-Muon Optimizercited by 14
  19. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 13
  20. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  21. 2025Cautious Weight Decaycited by 12
  22. 2025Dion2: A Simple Method to Shrink Matrix in Muoncited by 11
  23. 2026Insights on Muon from Simple Quadraticscited by 11
  24. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 11
  25. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 11
  26. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 11
  27. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  28. 2025Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Ordercited by 9
  29. 2025Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methodscited by 9
  30. 2026Improved Convergence Rates of Muon Optimizer for Nonconvex Optimizationcited by 9
  31. 2025Memory-Efficient LLM Pretraining via Minimalist Optimizer Designcited by 8
  32. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  33. 2025Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Ratescited by 8
  34. 2026Manifold constrained steepest descent for smooth and closed-set optimizationcited by 8
  35. 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancementcited by 8
  36. 2026Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decompositioncited by 8
  37. 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 8
  38. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 7

50 most cited of 177 All 177 (JSON)