How Much Orthogonalization Does Muon Need?
Not on the map. Why?
Abstract (arXiv)
Muon optimizers improve neural-network training by replacing ill-conditioned momentum updates with approximately semi-orthogonal updates. This motivates a practical question: how much orthogonalization does Muon actually require? We study this question using a relaxed cubic Newton--Schulz schedule derived directly for Muon's low precision singular value band. The resulting five-step cubic construction uses ten dominant matrix multiplications, compared with fifteen for five quintic Newton--Schulz iterations. The cubic schedule is not intended as a more accurate polar solver; instead, it is a principled low-cost variant that lets us probe the relation between polar accuracy, spectral shaping, and training quality. Across synthetic diagnostics, NanoGPT ablations, and training experiments on hybrid MoE/Mamba models, we find that training quality is not governed monotonically by polar-decomposition accuracy: truncated Polar Express, Muon-Jordan, cubic Newton--Schulz, and an explicit FP32 SVD polar factor can reach nearly indistinguishable final loss on GPT-2 Small, and cubic5 matches the Muon-Jordan quintic update within about 10-3 validation loss on hybrid MoE/Mamba models with one billion to four billion parameters. These results support cubic5 as a practical low-cost Muon orthogonalization variant, with empirical evidence of training-quality parity in the settings tested.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 1 of its 12 references that are in the map. First posted on arXiv 29 May 2026.
Builds on
1 of 12 references.
References · 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization
- 2026Muon2: Boosting Muon via Adaptive Second-Moment Preconditioning
- 2025Muon is Scalable for LLM Training
- 2026Orthogonal Witness Control for Muon Optimization via Sigmoid Spectral Reshaping
- 2026An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies
- 2025Effective Quantization of Muon Optimizer States
- 2026Muon-C: Operator-Aligned Muon for Convolutional Kernels
- 2026Move on Muon : A Hamiltonian probability gradient flow perspective of Muon optimizer