mlsanityData to 14 Oct 2026

The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm

arXiv 2505.16932 ↗First posted 22 May 2025cs.LG cs.AI cs.CLcited by 110

Not on the map. Why?

Abstract (arXiv)

Computing the polar decomposition and the related matrix sign function has been a well-studied problem in numerical analysis for decades. Recently, it has emerged as an important subroutine within the Muon optimizer for training deep neural networks. However, the requirements of this application differ sharply from classical settings: deep learning demands GPU-friendly algorithms that prioritize high throughput over high precision. We introduce Polar Express, a new method for computing the polar decomposition. Like Newton-Schulz and other classical polynomial methods, our approach uses only matrix-matrix multiplications, making it very efficient on GPUs. Inspired by earlier work of Chen & Chow and Nakatsukasa & Freund, Polar Express adapts the update rule at each iteration by solving a minimax optimization problem. We prove that this strategy minimizes error in a worst-case sense, allowing Polar Express to converge as rapidly as possible both in the early iterations and asymptotically. We also address finite-precision issues, making it practical to use in bfloat16. When integrated into Muon, our method yields consistent improvements in validation loss for a GPT-2 model trained on one to ten billion tokens from the FineWeb dataset, outperforming recent alternatives across a range of learning rates.

Key facts. Cited by 110 papers in the map (data through 14 Oct 2026). Builds on 11 of its 59 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

11 of 59 references.

References · 11

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  3. 2025Muon is Scalable for LLM Trainingcited by 428
  4. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  5. 2024Old Optimizer, New Norm: An Anthologycited by 193
  6. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  7. 2023A Spectral Condition for Feature Learningcited by 109
  8. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  9. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  10. 2024Modular Duality in Deep Learningcited by 77
  11. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 97.

110 papers.

Sort:

By area:

  1. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 55
  2. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 37
  3. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 35
  4. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 35
  5. 2026Controlled LLM Training on Spectral Spherecited by 31
  6. 2026Toto 2.0: Time Series Forecasting Enters the Scaling Eracited by 30
  7. 2025An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variantscited by 28
  8. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  9. 2026The Newton-Muon Optimizercited by 18
  10. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  11. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 17
  12. 2026Muon2: Boosting Muon via Adaptive Second-Moment Preconditioningcited by 17
38 more
  1. 2026Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimizationcited by 14
  2. 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 14
  3. 2026Insights on Muon from Simple Quadraticscited by 13
  4. 2026Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalizationcited by 12
  5. 2025Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methodscited by 11
  6. 2026Enhancing LLM Training via Spectral Clippingcited by 10
  7. 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 9
  8. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 8
  9. 2025Optimal Scaling Needs Optimal Normcited by 8
  10. 2025Optimizing Rank for High-Fidelity Implicit Neural Representationscited by 8
  11. 2026Manifold constrained steepest descent for smooth and closed-set optimizationcited by 8
  12. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  13. 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancementcited by 8
  14. 2026Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decompositioncited by 8
  15. 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 8
  16. 2026Can Muon Fine-tune Adam-Pretrained Models?cited by 7
  17. 2026TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Trainingcited by 6
  18. 2026MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-trainingcited by 6
  19. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  20. 2026Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformationcited by 6
  21. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 6
  22. 2026Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizerscited by 5
  23. 2026Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimizationcited by 5
  24. 2025Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimizationcited by 4
  25. 2026PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Trainingcited by 4
  26. 2026Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Trainingcited by 4
  27. 2026Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handlingcited by 4
  28. 2026LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifoldcited by 4
  29. 2026Dion3: Full-Stack Orthogonal Updatescited by 4
  30. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  31. 2026CacheMuon: Using Temporal Preconditioning To Approximate Polar Factorcited by 3
  32. 2026Muown Implicitly Performs Angular Step-size Decaycited by 3
  33. 2026DMuon: Efficient Distributed Muon Training with Near-Adam Overheadcited by 3
  34. 2026CMuon: Accelerating and Stabilizing Diffusion Transformer Training via Chunked Momentum Orthogonalizationcited by 3
  35. 2026SCORE: Replacing Layer Stacking with Contractive Recurrent Depthcited by 2
  36. 2026PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisationcited by 2
  37. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  38. 2026Tight Complexity of Muon under Heavy-Tailed Noisecited by 2

50 most cited of 110 All 110 (JSON)