mlsanityData to 14 Oct 2026

On the Convergence of Muon and Beyond

arXiv 2509.15816 ↗First posted 19 Sep 2025cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

The Muon optimizer has demonstrated strong empirical performance in training neural networks with matrix-structured parameters. A central theoretical question is how momentum variance reduction interacts with practical orthogonalization and weight decay. We analyze two variants: the one-batch Muon-MVR1 and the two-batch Muon-MVR2. To the best of our knowledge, this is the first horizon-free convergence analysis of variance-reduced Muon that jointly covers constant mini-batches, finite practical fixed-coefficient Newton--Schulz iterations, and decoupled weight decay in unconstrained stochastic nonconvex optimization. Muon-MVR1 and Muon-MVR2 attain anytime stationarity rates of widetildeO(T-1/4) and widetildeO(T-1/3), respectively, over T iterations. Under mean-square smoothness, the Muon-MVR2 rate matches the stochastic first-order lower bound up to logarithmic factors. The analysis provides alignment and norm bounds uniform in the step count, together with a nuclear-norm alignment bound that improves with the number of steps. Under the Polyak--Lojasiewicz (PL) condition and a uniform gradient bound along the iterates, Muon-MVR1 and Muon-MVR2 achieve last-iterate objective-gap rates of O(T-1/4) and O(T-1/3), respectively, with finite NS steps and decoupled weight decay under the prescribed horizon-free schedules. Experiments on CIFAR-10 and C4 support the practical effectiveness of the proposed variance-reduced Muon variants. Code is available at the hrefhttps://github.com/MaeChd/MUON-MVRMuon-MVR repository.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 53 of its 77 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

53 of 77 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2019On the Convergence of Adam and Beyondcited by 841
  6. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  7. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  8. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  9. 2025Muon is Scalable for LLM Trainingcited by 428
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  11. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  12. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
38 more
  1. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  2. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  3. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  4. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  5. 2024Old Optimizer, New Norm: An Anthologycited by 193
  6. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  7. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  8. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  9. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  10. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  11. 2020Momentum Improves Normalized SGDcited by 139
  12. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  13. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  14. 2018On the Convergence of Adaptive Gradient Methods for Nonconvex Optimizationcited by 91
  15. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  16. 2025A Note on the Convergence of Muoncited by 83
  17. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  18. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  19. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  20. 2025AdaMuon: Adaptive Muon Optimizercited by 67
  21. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  22. 2018Stochastic Nested Variance Reduction for Nonconvex Optimizationcited by 63
  23. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  24. 2025PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspectivecited by 52
  25. 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
  26. 2018Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximizationcited by 47
  27. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 47
  28. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  29. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  30. 2020Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentumcited by 40
  31. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  32. 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
  33. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  34. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 30
  35. 2019Linear Convergence of Adaptive Stochastic Gradient Descentcited by 29
  36. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
  37. 2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
  38. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 28.

31 papers.

Sort:

By area:

  1. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 37
  2. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 19
  3. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 17
  4. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 12
  5. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
  6. 2026Improved Convergence Rates of Muon Optimizer for Nonconvex Optimizationcited by 9
  7. 2026Muon Does Not Converge on Convex Lipschitz Functionscited by 9
  8. 2026Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decompositioncited by 8
  9. 2025Muon is Provably Faster with Momentum Variance Reductioncited by 7
  10. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 6
  11. 2026OLion: Approaching the Hadamard Ideal by Intersecting Spectral and ℓ∞ Implicit Biasescited by 3
  12. 2026Communication-Efficient Gluon in Federated Learningcited by 3
19 more
  1. 2026MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Modelscited by 3
  2. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  3. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2
  4. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  5. 2026Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Predictioncited by 2
  6. 2026MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimizationcited by 2
  7. 2026Scale Weight Decay and Train Bettercited by 2
  8. 2026Convergence guarantees for Muon: New parameter regimes and generalizationscited by 2
  9. 2026Dimension-Free Saddle-Point Escape in Muoncited by 1
  10. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  11. 2026OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimalitycited by 1
  12. 2026Muse: Representation Geometry of Muon Beyond Normalized Momentumcited by 1
  13. 2026OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimalitycited by 0
  14. 2026Accelerating LMO-Based Optimization via Implicit Gradient Transportcited by 0
  15. 2026Federated Compositional Muon Optimizer for Matrix-Wise Modelscited by 0
  16. 2026Nonsmooth Optimization via Orthogonalized Momentumcited by 0
  17. 2026Muon Under Gradient Noise and the Limits of Orthogonalization Near Optimacited by 0
  18. 2026AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Modelscited by 0
  19. 2026Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalizationcited by 0