mlsanityData to 14 Oct 2026

Preconditioning Benefits of Spectral Orthogonalization in Muon

arXiv 2601.13474 ↗First posted 20 Jan 2026cs.LG cs.AI math.OCcited by 30

Not on the map. Why?

Abstract (arXiv)

The Muon optimizer, a matrix-structured algorithm that leverages spectral orthogonalization of gradients, is a milestone in the pretraining of large language models. However, the underlying mechanisms of Muon -- particularly the role of gradient orthogonalization -- remain poorly understood, with very few works providing end-to-end analyses that rigorously explain its advantages in concrete applications. We take a step by studying the effectiveness of a simplified variant of Muon through two case studies: matrix factorization, and in-context learning of linear transformers. For both problems, we prove that simplified Muon converges linearly with iteration complexities independent of the relevant condition number, provably outperforming gradient descent and Adam. Our analysis reveals that the Muon dynamics decouple into a collection of independent scalar sequences in the spectral domain, each exhibiting similar convergence behavior. Our theory formalizes the preconditioning effect induced by spectral orthogonalization, offering insight into Muon's effectiveness in these matrix optimization problems and potentially beyond.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 20 Jan 2026.

Builds on

42 of 56 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  5. 2025Muon is Scalable for LLM Trainingcited by 428
  6. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  7. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  8. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  9. 2024Old Optimizer, New Norm: An Anthologycited by 193
  10. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  11. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 160
  12. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
30 more
  1. 2023In-Context Convergence of Transformerscited by 115
  2. 2020Spectral Methods for Data Science: A Statistical Perspectivecited by 112
  3. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  4. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  5. 2025A Note on the Convergence of Muoncited by 83
  6. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  7. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 78
  8. 2024Modular Duality in Deep Learningcited by 77
  9. 2020Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descentcited by 71
  10. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  11. 2025PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspectivecited by 52
  12. 2021Acceleration Methodscited by 48
  13. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  14. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  15. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  16. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 37
  17. 2023The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensingcited by 33
  18. 2021On the computational and statistical complexity of over-parameterized matrix sensingcited by 31
  19. 2025When do spectral gradient updates help in deep learning?cited by 31
  20. 2020Low-Rank Matrix Recovery with Scaled Subgradient Methods: Fast and Robust Convergence Without the Condition Numbercited by 28
  21. 2022Orthogonalising gradients to speed up neural network optimisationcited by 27
  22. 2022Preconditioned Gradient Descent for Overparameterized Nonconvex Burer--Monteiro Factorization with Global Optimality Certificationcited by 21
  23. 2024Towards Quantifying the Preconditioning Effect of Adamcited by 19
  24. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 19
  25. 2023How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initializationcited by 18
  26. 2025On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learningcited by 12
  27. 2025Preconditioned Gradient Descent for Over-Parameterized Nonconvex Matrix Factorizationcited by 12
  28. 2025Muon Optimizer Accelerates Grokkingcited by 12
  29. 2024A Theoretical Analysis of Self-Supervised Learning for Vision Transformerscited by 6
  30. 2024Provable Acceleration of Nesterov's Accelerated Gradient for Rectangular Matrix Factorization and Linear Neural Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 30.

30 papers.

Sort:

By area:

  1. 2026The Newton-Muon Optimizercited by 18
  2. 2026Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memorycited by 14
  3. 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 14
  4. 2026Insights on Muon from Simple Quadraticscited by 13
  5. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  6. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 12
  7. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
  8. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 6
  9. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  10. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  11. 2026Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimizationcited by 5
  12. 2026Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimizationcited by 4
18 more
  1. 2026Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Trainingcited by 4
  2. 2026AMUSE: Anytime Muon with Stable Gradient Evaluationcited by 4
  3. 2026DynMuon: A Dynamic Spectral Shaping View of Muoncited by 3
  4. 2026MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Modelscited by 3
  5. 2026Muon Learns More Robust and Transferable Features than Adamcited by 3
  6. 2026Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamicscited by 3
  7. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2
  8. 2026Tight Complexity of Muon under Heavy-Tailed Noisecited by 2
  9. 2026Convergence of Spectral Descent for Non-smooth Optimizationcited by 2
  10. 2026Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filteringcited by 2
  11. 2026Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspectivecited by 2
  12. 2026Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimizationcited by 1
  13. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  14. 2026Reassessing Muon for Matrix Factorizationcited by 1
  15. 2026When Does Muon Help Agentic Reinforcement Learning?cited by 1
  16. 2026On the Principles Behind Neural Network Optimizerscited by 0
  17. 2026Convergence of Practical Muoncited by 0
  18. 2026The Row Normalization Puzzle in Muoncited by 0