mlsanityData to 14 Oct 2026

On the Convergence Analysis of Muon

arXiv 2505.23737 ↗First posted 29 May 2025stat.ML cs.IT cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

The majority of parameters in neural networks are naturally represented as matrices. However, most commonly used optimizers treat these matrix parameters as flattened vectors during optimization, potentially overlooking their inherent structural properties. Recently, an optimizer called Muon has been proposed, specifically designed to optimize matrix-structured parameters. Extensive empirical evidence shows that Muon can significantly outperform traditional optimizers when training neural networks. Nonetheless, the theoretical understanding of Muon's convergence behavior and the reasons behind its superior performance remain limited. In this work, we present a comprehensive convergence rate analysis of Muon and its comparison with Gradient Descent (GD). We characterize the conditions under which Muon can outperform GD. Our theoretical results reveal that Muon can benefit from the low-rank structure of Hessian matrices, a phenomenon widely observed in practical neural network training. Our experimental results support and corroborate the theoretical findings.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

30 of 50 references.

References · 30

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  3. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  4. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  5. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  6. 2025Muon is Scalable for LLM Trainingcited by 428
  7. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  8. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  9. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  10. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  11. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  12. 2024Old Optimizer, New Norm: An Anthologycited by 193
18 more
  1. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  2. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  3. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  4. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  5. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  6. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  7. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  8. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  9. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  10. 2025A Note on the Convergence of Muoncited by 83
  11. 2024A New Perspective on Shampoo's Preconditionercited by 59
  12. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  13. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  14. 2019SGD Converges to Global Minimum in Deep Learning via Star-convex Pathcited by 39
  15. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 33
  16. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 30
  17. 2021Tensor Normal Training for Deep Learning Modelscited by 24
  18. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memorycited by 14
  2. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
  3. 2026Muon Does Not Converge on Convex Lipschitz Functionscited by 9
  4. 2026Phases of Muon: When Muon Eclipses SignSGDcited by 3
  5. 2026PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Trainingcited by 1