mlsanityData to 14 Oct 2026

A Note on the Convergence of Muon

arXiv 2502.02900 ↗First posted 5 Feb 2025math.OCcited by 83

Not on the map. Why?

Abstract (arXiv)

In this note, we inspect the convergence of a new optimizer for pretraining LLMs, namely the Muon optimizer. Such an optimizer is closely related to a specialized steepest descent method where the update direction is the minimizer of the quadratic approximation of the objective function under spectral norm. We provide the convergence analysis on both versions of the optimizer and discuss its implications.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Feb 2025.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 31; 2026: 52.

83 papers.

Sort:

By area:

  1. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  2. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  3. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  4. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  5. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  6. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  7. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 37
  8. 2026Convergence of Muon with Newton-Schulzcited by 36
  9. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 35
  10. 2025On the Convergence of Muon and Beyondcited by 31
  11. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 30
  12. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
38 more
  1. 2025An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variantscited by 28
  2. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  3. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23
  4. 2025LiMuon: Light and Fast Muon Optimizer for Large Modelscited by 22
  5. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 19
  6. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 19
  7. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 18
  8. 2026The Newton-Muon Optimizercited by 18
  9. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  10. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 17
  11. 2025ROOT: Robust Orthogonalized Optimizer for Neural Network Trainingcited by 15
  12. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 14
  13. 2026Insights on Muon from Simple Quadraticscited by 13
  14. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  15. 2026FISMO: Fisher-Structured Momentum-Orthogonalized Optimizercited by 11
  16. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
  17. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  18. 2025Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Ordercited by 9
  19. 2026Improved Convergence Rates of Muon Optimizer for Nonconvex Optimizationcited by 9
  20. 2026Muon Does Not Converge on Convex Lipschitz Functionscited by 9
  21. 2026Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decompositioncited by 8
  22. 2025On Provable Benefits of Muon in Federated Learningcited by 7
  23. 2025Unbiased Gradient Low-Rank Projectioncited by 7
  24. 2025MuonAll: Muon Variant for Efficient Finetuning of Large Language Modelscited by 7
  25. 2025Muon is Provably Faster with Momentum Variance Reductioncited by 7
  26. 2026Stabilizing Native Low-Rank LLM Pretrainingcited by 6
  27. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  28. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  29. 2025On the Convergence Analysis of Muoncited by 5
  30. 2026Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergencecited by 5
  31. 2026Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimizationcited by 5
  32. 2025Non-Euclidean Broximal Point Method: A Blueprint for Geometry-Aware Optimizationcited by 4
  33. 2025DeMuon: A Decentralized Muon for Matrix Optimization over Graphscited by 4
  34. 2025Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimizationcited by 4
  35. 2025Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Trainingcited by 3
  36. 2026Communication-Efficient Gluon in Federated Learningcited by 3
  37. 2026MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Modelscited by 3
  38. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3

50 most cited of 83 All 83 (JSON)