mlsanityData to 14 Oct 2026

Convergence of Muon with Newton-Schulz

arXiv 2601.19156 ↗First posted 27 Jan 2026stat.ML cs.LG math.OCcited by 36

Not on the map. Why?

Abstract (arXiv)

We analyze Muon as originally proposed and used in practice -- using the momentum orthogonalization with a few Newton-Schulz steps. The prior theoretical results replace this key step in Muon with an exact SVD-based polar factor. We prove that Muon with Newton-Schulz converges to a stationary point at the same rate as the SVD-polar idealization, up to a constant factor for a given number q of Newton-Schulz steps. We further analyze this constant factor and prove that it converges to 1 doubly exponentially in q and improves with the degree of the polynomial used in Newton-Schulz for approximating the orthogonalization direction. We also prove that Muon removes the typical square-root-of-rank loss compared to its vector-based counterpart, SGD with momentum. Our results explain why Muon with a few low-degree Newton-Schulz steps matches exact-polar (SVD) behavior at a much faster wall-clock time and explain how much momentum matrix orthogonalization via Newton-Schulz benefits over the vector-based optimizer. Overall, our theory justifies the practical Newton-Schulz design of Muon, narrowing its practice-theory gap.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 18 of its 32 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

18 of 32 references.

References · 18

  1. 2021Convex optimizationcited by 2,515
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  3. 2019On the Convergence of Adam and Beyondcited by 841
  4. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  5. 2025Muon is Scalable for LLM Trainingcited by 428
  6. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  7. 2024Old Optimizer, New Norm: An Anthologycited by 193
  8. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  9. 2018A Sufficient Condition for Convergences of Adam and RMSPropcited by 126
  10. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  11. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  12. 2025A Note on the Convergence of Muoncited by 83
6 more
  1. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  2. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  3. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  4. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 30
  5. 2022Orthogonalising gradients to speed up neural network optimisationcited by 27
  6. 2025Muon Optimizer Accelerates Grokkingcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 36.

36 papers.

Sort:

By area:

  1. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 17
  2. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 15
  3. 2026Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memorycited by 14
  4. 2026Insights on Muon from Simple Quadraticscited by 13
  5. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  6. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  7. 2026Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimizationcited by 5
  8. 2026Aurora: A Leverage-Aware Spectral Optimizercited by 4
  9. 2026MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Modelscited by 3
  10. 2026CacheMuon: Using Temporal Preconditioning To Approximate Polar Factorcited by 3
  11. 2026Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizerscited by 2
  12. 2026DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentumcited by 2
24 more
  1. 2026Tight Complexity of Muon under Heavy-Tailed Noisecited by 2
  2. 2026Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Lawscited by 2
  3. 2026Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspectivecited by 2
  4. 2026Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimizationcited by 2
  5. 2026Convergence guarantees for Muon: New parameter regimes and generalizationscited by 2
  6. 2026Muon in Vision Transformers: Optimizer-Recipe Interactions and Gradient Spectracited by 1
  7. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  8. 2026OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimalitycited by 1
  9. 2026Reassessing Muon for Matrix Factorizationcited by 1
  10. 2026Muse: Representation Geometry of Muon Beyond Normalized Momentumcited by 1
  11. 2026Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentumcited by 1
  12. 2026OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimalitycited by 0
  13. 2026Accelerating LMO-Based Optimization via Implicit Gradient Transportcited by 0
  14. 2026Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuningcited by 0
  15. 2026Federated Compositional Muon Optimizer for Matrix-Wise Modelscited by 0
  16. 2026Nonsmooth Optimization via Orthogonalized Momentumcited by 0
  17. 2026Muon Under Gradient Noise and the Limits of Orthogonalization Near Optimacited by 0
  18. 2026Cost-free Spectral Estimation for Adaptive Newton--Schulz in Matrix Optimizerscited by 0
  19. 2026Convergence of Practical Muoncited by 0
  20. 2026The Row Normalization Puzzle in Muoncited by 0
  21. 2026Inner Momentum for Differentially Private Muoncited by 0
  22. 2026Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalizationcited by 0
  23. 2026ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Trainingcited by 0
  24. 2026From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Biascited by 0