# Preconditioning Benefits of Spectral Orthogonalization in Muon

arXiv: https://arxiv.org/abs/2601.13474 · HTML: https://mlsanity.com/paper/2601.13474/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 56 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [Trained Transformers Learn Linear Models In-Context](https://mlsanity.com/paper/2306.09927/) (2023)
- [Transformers learn to implement preconditioned gradient descent for in-context learning](https://mlsanity.com/paper/2306.00297/) (2023)
- [Old Optimizer, New Norm: An Anthology](https://mlsanity.com/paper/2409.20325/) (2024)
- [Training Deep Learning Models with Norm-Constrained LMOs](https://mlsanity.com/paper/2502.07529/) (2025)
- [Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview](https://mlsanity.com/paper/1809.09573/) (2018)
- [Why Transformers Need Adam: A Hessian Perspective](https://mlsanity.com/paper/2402.16788/) (2024)
- [In-Context Convergence of Transformers](https://mlsanity.com/paper/2310.05249/) (2023)
- [Spectral Methods for Data Science: A Statistical Perspective](https://mlsanity.com/paper/2012.08496/) (2020)
- [Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization](https://mlsanity.com/paper/2503.12645/) (2025)
- [Muon Optimizes Under Spectral Norm Constraints](https://mlsanity.com/paper/2506.15054/) (2025)
- [A Note on the Convergence of Muon](https://mlsanity.com/paper/2502.02900/) (2025)
- [Practical Efficiency of Muon for Pretraining](https://mlsanity.com/paper/2505.02222/) (2025)
- [Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction](https://mlsanity.com/paper/2106.15013/) (2021)
- [Modular Duality in Deep Learning](https://mlsanity.com/paper/2410.21265/) (2024)
- [Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent](https://mlsanity.com/paper/2005.08898/) (2020)
- [ASGO: Adaptive Structured Gradient Optimization](https://mlsanity.com/paper/2503.20762/) (2025)
- [PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective](https://mlsanity.com/paper/2505.21799/) (2025)
- [Acceleration Methods](https://mlsanity.com/paper/2101.09545/) (2021)
- [Muon Outperforms Adam in Tail-End Associative Memory Learning](https://mlsanity.com/paper/2509.26030/) (2025)
- [Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data](https://mlsanity.com/paper/2502.04664/) (2025)
- [In-Context Learning with Representations: Contextual Generalization of Trained Transformers](https://mlsanity.com/paper/2408.10147/) (2024)
- [Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?](https://mlsanity.com/paper/2511.00674/) (2025)
- [The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensing](https://mlsanity.com/paper/2302.01186/) (2023)
- [On the computational and statistical complexity of over-parameterized matrix sensing](https://mlsanity.com/paper/2102.02756/) (2021)

## Built on by (most cited first, 30 of 30)

- [The Newton-Muon Optimizer](https://mlsanity.com/paper/2604.01472/) (2026)
- [Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory](https://mlsanity.com/paper/2603.26554/) (2026)
- [Muon is Not That Special: Random or Inverted Spectra Work Just as Well](https://mlsanity.com/paper/2605.11181/) (2026)
- [Insights on Muon from Simple Quadratics](https://mlsanity.com/paper/2602.11948/) (2026)
- [On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer](https://mlsanity.com/paper/2603.09952/) (2026)
- [HTMuon: Improving Muon via Heavy-Tailed Spectral Correction](https://mlsanity.com/paper/2603.10067/) (2026)
- [Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers](https://mlsanity.com/paper/2605.18106/) (2026)
- [Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval](https://mlsanity.com/paper/2601.22652/) (2026)
- [Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds](https://mlsanity.com/paper/2605.09238/) (2026)
- [Why Muon Outperforms Adam: A Curvature Perspective](https://mlsanity.com/paper/2606.04662/) (2026)
- [Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/2608.26288/) (2026)
- [Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization](https://mlsanity.com/paper/2602.06880/) (2026)
- [Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training](https://mlsanity.com/paper/2603.17970/) (2026)
- [AMUSE: Anytime Muon with Stable Gradient Evaluation](https://mlsanity.com/paper/2605.22432/) (2026)
- [DynMuon: A Dynamic Spectral Shaping View of Muon](https://mlsanity.com/paper/2605.17109/) (2026)
- [MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models](https://mlsanity.com/paper/2605.19619/) (2026)
- [Muon Learns More Robust and Transferable Features than Adam](https://mlsanity.com/paper/2606.09658/) (2026)
- [Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics](https://mlsanity.com/paper/2606.30509/) (2026)
- [When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Bounds](https://mlsanity.com/paper/2605.06615/) (2026)
- [Tight Complexity of Muon under Heavy-Tailed Noise](https://mlsanity.com/paper/2605.18528/) (2026)
- [Convergence of Spectral Descent for Non-smooth Optimization](https://mlsanity.com/paper/2605.26977/) (2026)
- [Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering](https://mlsanity.com/paper/2606.03899/) (2026)
- [Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective](https://mlsanity.com/paper/2606.21514/) (2026)
- [Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimization](https://mlsanity.com/paper/2604.04231/) (2026)
- [A Note on Stability for Orthogonalized Matrix Momentum with Client Sampling](https://mlsanity.com/paper/2606.01720/) (2026)
- [Reassessing Muon for Matrix Factorization](https://mlsanity.com/paper/2607.13246/) (2026)
- [When Does Muon Help Agentic Reinforcement Learning?](https://mlsanity.com/paper/2607.16169/) (2026)
- [On the Principles Behind Neural Network Optimizers](https://mlsanity.com/paper/2608.16760/) (2026)
- [Convergence of Practical Muon](https://mlsanity.com/paper/2609.33152/) (2026)
- [The Row Normalization Puzzle in Muon](https://mlsanity.com/paper/2609.39114/) (2026)

## Abstract (arXiv)

The Muon optimizer, a matrix-structured algorithm that leverages spectral orthogonalization of gradients, is a milestone in the pretraining of large language models. However, the underlying mechanisms of Muon -- particularly the role of gradient orthogonalization -- remain poorly understood, with very few works providing end-to-end analyses that rigorously explain its advantages in concrete applications. We take a step by studying the effectiveness of a simplified variant of Muon through two case studies: matrix factorization, and in-context learning of linear transformers. For both problems, we prove that simplified Muon converges linearly with iteration complexities independent of the relevant condition number, provably outperforming gradient descent and Adam. Our analysis reveals that the Muon dynamics decouple into a collection of independent scalar sequences in the spectral domain, each exhibiting similar convergence behavior. Our theory formalizes the preconditioning effect induced by spectral orthogonalization, offering insight into Muon's effectiveness in these matrix optimization problems and potentially beyond.