mlsanityData to 14 Oct 2026

The Spectral Dynamics and Noise Geometry of Muon

arXiv 2606.08388 ↗First posted 7 Jun 2026cs.LG math.OC stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Muon replaces a matrix gradient G=USigma V^top by its polar factor UV^top. This keeps the singular directions selected by the gradient, but makes the update spectrum flat. We study the optimization bias created by this operation. Under explicit alignment assumptions, we prove that the polar update is the one-step entropy-maximizing choice among bounded updates that use the gradient singular directions and do not adapt to the current weight spectrum. In an underdetermined regression model, we derive exact singular-value dynamics for continuous-time Muon and identify a measurement-dependent condition under which the normalized spectrum moves toward equal nonzero singular values. This geometry also rules out a common low-rank interpretation: at fixed Frobenius norm, Muon's distinguished state has a flat spectrum, whereas nuclear-norm minimization favors spectral concentration. Controlled matrix-sensing experiments separate the effect from simple gradient rescaling, show that norm-matched gradient descent does not reproduce Muon, and recover the predicted flattening trend across broad ablations. In small NanoGPT pretraining, Muon preserves stable rank, has a broad learning-rate plateau, and improves validation loss relative to AdamW; in a matched small-ViT control, the ranking reverses. The resulting picture is regime-dependent: Muon is not universally superior, but its flat-spectrum bias can help when many spectral directions need to remain active.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 7 Jun 2026.

Builds on

30 of 41 references.

References · 30

  1. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  2. 2017Implicit Regularization in Matrix Factorizationcited by 322
  3. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  4. 2018An Empirical Model of Large-Batch Trainingcited by 281
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  6. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  7. 2025A Note on the Convergence of Muoncited by 83
  8. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  9. 2025PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspectivecited by 52
  10. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  11. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  12. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
18 more
  1. 2022Orthogonalising gradients to speed up neural network optimisationcited by 27
  2. 2025Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Trainingcited by 23
  3. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 19
  4. 2026Muon2: Boosting Muon via Adaptive Second-Moment Preconditioningcited by 17
  5. 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 14
  6. 2021Implicit Regularization in Matrix Sensing via Mirror Descentcited by 11
  7. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 11
  8. 2026Uniform Spectral Growth and Convergence of Muon in LoRA-Style Matrix Factorizationcited by 9
  9. 2026Muon Does Not Converge on Convex Lipschitz Functionscited by 9
  10. 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 8
  11. 2026TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizerscited by 6
  12. 2026Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfercited by 6
  13. 2026Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergencecited by 5
  14. 2026AMUSE: Anytime Muon with Stable Gradient Evaluationcited by 4
  15. 2026pAI/MSc: ML Theory Research with Humans on the Loopcited by 2
  16. 2026DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentumcited by 2
  17. 2026OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scalingcited by 1
  18. 2026When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Studycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026The Loss Does Not See the Basis, but Adam Doescited by 0
  2. 2026Physical Muon: Orthogonalization as an Equilibrium Computationcited by 0
  3. 2026Does Muon Need Fine-Grained Spectral Shaping?cited by 0