mlsanityData to 14 Oct 2026

Muon Optimizes Under Spectral Norm Constraints

arXiv 2506.15054 ↗First posted 18 Jun 2025cs.LG math.OC stat.MLcited by 86

Not on the map. Why?

Abstract (arXiv)

The pursuit of faster optimization algorithms remains an active and important research direction in deep learning. Recently, the Muon optimizer [JJB+24] has demonstrated promising empirical performance, but its theoretical foundation remains less understood. In this paper, we bridge this gap and provide a theoretical analysis of Muon by placing it within the Lion-K family of optimizers [CLLL24]. Specifically, we show that Muon corresponds to Lion-K when equipped with the nuclear norm, and we leverage the theoretical results of Lion-K to establish that Muon (with decoupled weight decay) implicitly solves an optimization problem that enforces a constraint on the spectral norm of weight matrices. This perspective not only demystifies the implicit regularization effects of Muon but also leads to natural generalizations through varying the choice of convex map K, allowing for the exploration of a broader class of implicitly regularized and constrained optimization algorithms.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 24 of its 44 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

24 of 44 references.

References · 24

  1. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  2. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  3. 2025Muon is Scalable for LLM Trainingcited by 428
  4. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  5. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  6. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  7. 2024Old Optimizer, New Norm: An Anthologycited by 193
  8. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  9. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  10. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  11. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  12. 2025A Note on the Convergence of Muoncited by 83
12 more
  1. 2017Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descentcited by 66
  2. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  3. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  4. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  5. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  6. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  7. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  8. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  9. 2024Curvature-Informed SGD via General Purpose Lie-Group Preconditionerscited by 16
  10. 2024DeMo: Decoupled Momentum Optimizationcited by 11
  11. 2022Black Box Lie Group Preconditioners for SGDcited by 9
  12. 2022The Iterates of the Frank-Wolfe Algorithm May Not Convergecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 71.

86 papers.

Sort:

By area:

  1. 2025AdaMuon: Adaptive Muon Optimizercited by 67
  2. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 37
  3. 2026Convergence of Muon with Newton-Schulzcited by 36
  4. 2025On the Convergence of Muon and Beyondcited by 31
  5. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 30
  6. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
  7. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  8. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23
  9. 2025LiMuon: Light and Fast Muon Optimizer for Large Modelscited by 22
  10. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 19
  11. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 18
  12. 2026The Newton-Muon Optimizercited by 18
38 more
  1. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 17
  2. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 16
  3. 2026Training-Free Looped Transformerscited by 15
  4. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 14
  5. 2026Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memorycited by 14
  6. 2025Cautious Weight Decaycited by 13
  7. 2026Insights on Muon from Simple Quadraticscited by 13
  8. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  9. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 12
  10. 2026FISMO: Fisher-Structured Momentum-Orthogonalized Optimizercited by 11
  11. 2025Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Ratescited by 8
  12. 2025On Provable Benefits of Muon in Federated Learningcited by 7
  13. 2025MuonAll: Muon Variant for Efficient Finetuning of Large Language Modelscited by 7
  14. 2026Can Muon Fine-tune Adam-Pretrained Models?cited by 7
  15. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 6
  16. 2026Stabilizing Native Low-Rank LLM Pretrainingcited by 6
  17. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  18. 2026Muown: Row-Norm Control for Muon Optimizationcited by 6
  19. 2026Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformationcited by 6
  20. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  21. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5
  22. 2026Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizerscited by 5
  23. 2026Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learningcited by 5
  24. 2026Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergencecited by 5
  25. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 5
  26. 2026Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimizationcited by 5
  27. 2025DeMuon: A Decentralized Muon for Matrix Optimization over Graphscited by 4
  28. 2026Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimizationcited by 4
  29. 2026Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handlingcited by 4
  30. 2026Momentum Guidance: Plug-and-Play Guidance for Flow Modelscited by 3
  31. 2026DynMuon: A Dynamic Spectral Shaping View of Muoncited by 3
  32. 2026Move on Muon : A Hamiltonian probability gradient flow perspective of Muon optimizercited by 3
  33. 2026Muon Learns More Robust and Transferable Features than Adamcited by 3
  34. 2026CacheMuon: Using Temporal Preconditioning To Approximate Polar Factorcited by 3
  35. 2026NS-RGS: Newton-Schulz based Riemannian gradient method for orthogonal group synchronizationcited by 2
  36. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2
  37. 2026Tight Complexity of Muon under Heavy-Tailed Noisecited by 2
  38. 2026LionMuon: Alternating Spectral and Sign Descent for Efficient Trainingcited by 2

50 most cited of 86 All 86 (JSON)