# Uniform Spectral Growth and Convergence of Muon in LoRA-Style Matrix Factorization

arXiv: https://arxiv.org/abs/2602.06385 · HTML: https://mlsanity.com/paper/2602.06385/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2026.

## Built on by (most cited first, 9 of 9)

- [Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds](https://mlsanity.com/paper/2605.09238/) (2026)
- [AMUSE: Anytime Muon with Stable Gradient Evaluation](https://mlsanity.com/paper/2605.22432/) (2026)
- [The Spectral Dynamics and Noise Geometry of Muon](https://mlsanity.com/paper/2606.08388/) (2026)
- [Muon Learns More Robust and Transferable Features than Adam](https://mlsanity.com/paper/2606.09658/) (2026)
- [Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics](https://mlsanity.com/paper/2606.30509/) (2026)
- [Unsupervised Adaptation of PDE Foundation Models](https://mlsanity.com/paper/2608.07053/) (2026)
- [Approximate Muon with low-rank adapters](https://mlsanity.com/paper/2608.14492/) (2026)
- [The Loss Does Not See the Basis, but Adam Does](https://mlsanity.com/paper/2608.05136/) (2026)
- [Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization](https://mlsanity.com/paper/2610.04249/) (2026)

## Abstract (arXiv)

Spectral gradient descent (SpecGD) orthogonalizes the matrix parameter updates and has inspired practical optimizers such as Muon. They often perform well in large language model (LLM) training, but their dynamics remain poorly understood. In the low-rank adaptation (LoRA) setting, where weight updates are parameterized as a product of two low-rank factors, we find a distinctive spectral phenomenon under Muon in LoRA fine-tuning of LLMs: singular values of the LoRA product show near-uniform growth across the spectrum, despite orthogonalization being performed on the two factors separately. Motivated by this observation, we analyze spectral gradient flow (SpecGF)-a continuous-time analogue of SpecGD-in a simplified LoRA-style matrix factorization setting and prove "equal-rate" dynamics: all singular values grow at equal rates up to small deviations. Consequently, smaller singular values attain their target values earlier than larger ones, sharply contrasting with the largest-first stepwise learning observed in standard gradient flow. Moreover, we prove that SpecGF in our setting converges to global minima from almost all initializations, provided the factor norms remain bounded; with ℓ2 regularization, we obtain global convergence. Lastly, we corroborate our theory with experiments in the same setting.