# Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training

arXiv: https://arxiv.org/abs/2603.17970 · HTML: https://mlsanity.com/paper/2603.17970/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 18 Mar 2026.

## Builds on (most cited first, 17 of 17)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [SOAP: Improving and Stabilizing Shampoo using Adam](https://mlsanity.com/paper/2409.11321/) (2024)
- [The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm](https://mlsanity.com/paper/2505.16932/) (2025)
- [Optimal whitening and decorrelation](https://mlsanity.com/paper/1512.00809/) (2015)
- [NorMuon: Making Muon more efficient and scalable](https://mlsanity.com/paper/2510.05491/) (2025)
- [AdaMuon: Adaptive Muon Optimizer](https://mlsanity.com/paper/2507.11005/) (2025)
- [PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective](https://mlsanity.com/paper/2505.21799/) (2025)
- [Preconditioning Benefits of Spectral Orthogonalization in Muon](https://mlsanity.com/paper/2601.13474/) (2026)
- [Beyond the Ideal: Analyzing the Inexact Muon Update](https://mlsanity.com/paper/2510.19933/) (2025)
- [A Stable Whitening Optimizer for Efficient Neural Network Training](https://mlsanity.com/paper/2506.07254/) (2025)
- [94% on CIFAR-10 in 3.29 Seconds on a Single GPU](https://mlsanity.com/paper/2404.00498/) (2024)
- [What Really Matters in Matrix-Whitening Optimizers?](https://mlsanity.com/paper/2510.25000/) (2025)
- [On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regime](https://mlsanity.com/paper/2601.03162/) (2026)

## Built on by (most cited first, 4 of 4)

- [Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization](https://mlsanity.com/paper/2606.27216/) (2026)
- [Muon in Vision Transformers: Optimizer-Recipe Interactions and Gradient Spectra](https://mlsanity.com/paper/2605.24770/) (2026)
- [Muonp: Muon with Fractional Spectral Powers](https://mlsanity.com/paper/2606.13867/) (2026)
- [Relative Generalization Invariance of LLM Pretraining](https://mlsanity.com/paper/2609.33016/) (2026)

## Abstract (arXiv)

Orthogonalized-momentum optimizers such as Muon improve transformer training by approximately whitening/orthogonalizing matrix-valued momentum updates via a short polar-decomposition iteration. However, polar-factor approximations typically require multiple large matrix multiplications, and the resulting overhead can be substantial and hardware-dependent. We introduce MUD (MomentUm Decorrelation), a complementary whitening approach that replaces Muon's polar update with a triangular (Cholesky-like) whitening surrogate inspired by classical Gram--Schmidt and Gauss-Seidel ideas. We show that row-orthonormal matrices are fixed points of the MUD map, relate the inner step to symmetric Gauss-Seidel preconditioning of the Gram matrix, and prove quadratic local convergence near the fixed point. In terms of time-to-perplexity, MUD yields consistent 10-50% wall-clock improvements over tuned AdamW and Muon in time-to-perplexity, typically converging slightly slower per step than Muon but with substantially lower optimizer overhead -- relative to Muon, MUD improves peak tokens/s by roughly 1.3-2.6× across most settings and up to nearly 3× on GPT-2 large on an A100. We also demonstrate training a ESM-2 150M protein language model, where MUD matches Muon-level validation perplexity in significantly less wall-clock time.