mlsanityData to 14 Oct 2026

MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration

arXiv 2603.28254 ↗First posted 30 Mar 2026cs.LG stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

Orthogonalized-update optimizers such as Muon improve training of matrix-valued parameters, but existing extensions typically either rescale updates after orthogonalization or use heavier whitening-based preconditioners before it. We introduce method, a lightweight family of pre-orthogonalization equilibration schemes for Muon with three forms: two-sided row/column normalization (RC), row normalization (R), and column normalization (C). By rebalancing the momentum matrix before finite-step Newton--Schulz orthogonalization, method improves the geometry seen by orthogonalization. We show that finite-step orthogonalization is governed by the input spectrum, especially stable rank and condition number, and that row/column normalization acts as a zeroth-order surrogate for whitening. For hidden matrix weights, R is the default variant. Theoretically, method (R) retains the standard widetildemathcal O(T-1/4) Muon-type nonconvex stationarity guarantee with decoupled weight decay and a horizon-free diminishing learning-rate schedule, and extends it to finite-step NS5 up to an explicit inexactness constant. In LLaMA2 pretraining on C4, method (R) consistently outperforms Muon on 130M, 350M, and 1B models, with faster convergence and lower validation perplexity. The code is available at the hrefhttps://github.com/MaeChd/muon-eqMuonEq codebase.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 48 of its 61 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

48 of 61 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  7. 2025Muon is Scalable for LLM Trainingcited by 428
  8. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  9. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  10. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
  11. 2024Old Optimizer, New Norm: An Anthologycited by 193
  12. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
36 more
  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  2. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  3. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 110
  4. 2023Convergence of Adam Under Relaxed Assumptionscited by 99
  5. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  6. 2025NorMuon: Making Muon more efficient and scalablecited by 84
  7. 2025A Note on the Convergence of Muoncited by 83
  8. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  9. 2025Dion: Distributed Orthonormalized Updatescited by 68
  10. 2025AdaMuon: Adaptive Muon Optimizercited by 67
  11. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  12. 2018Stochastic Nested Variance Reduction for Nonconvex Optimizationcited by 63
  13. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  14. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  15. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  16. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  17. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  18. 2026Convergence of Muon with Newton-Schulzcited by 36
  19. 2025On the Convergence of Muon and Beyondcited by 31
  20. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
  21. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  22. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23
  23. 2025LiMuon: Light and Fast Muon Optimizer for Large Modelscited by 22
  24. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 18
  25. 2023Sketchy: Memory-efficient Adaptive Regularization with Frequent Directionscited by 17
  26. 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 17
  27. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 14
  28. 2025Memory-Efficient LLM Pretraining via Minimalist Optimizer Designcited by 13
  29. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  30. 2025Effective Quantization of Muon Optimizer Statescited by 11
  31. 2026FISMO: Fisher-Structured Momentum-Orthogonalized Optimizercited by 11
  32. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  33. 2026Improved Convergence Rates of Muon Optimizer for Nonconvex Optimizationcited by 9
  34. 2025On the O(frac√dK1/4) Convergence Rate of AdamW Measured by ℓ1 Normcited by 8
  35. 2025MuonAll: Muon Variant for Efficient Finetuning of Large Language Modelscited by 7
  36. 2025Muon is Provably Faster with Momentum Variance Reductioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
  2. 2026Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoningcited by 7
  3. 2026Aurora: A Leverage-Aware Spectral Optimizercited by 4
  4. 2026PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisationcited by 2
  5. 2026Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Predictioncited by 2
  6. 2026MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimizationcited by 2
  7. 2026Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimizationcited by 2
  8. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  9. 2026Muse: Representation Geometry of Muon Beyond Normalized Momentumcited by 1
  10. 2026MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioningcited by 1
  11. 2026Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvaturecited by 1
  12. 2026From SGD to Muon: Adaptive Optimization via Schatten-p Normscited by 0
5 more
  1. 2026Blog: Survey of Optimizerscited by 0
  2. 2026Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Trainingcited by 0
  3. 2026The Row Normalization Puzzle in Muoncited by 0
  4. 2026Optimizing Large Language Models with Chained LMOscited by 0
  5. 2026From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muoncited by 0