# MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration

arXiv: https://arxiv.org/abs/2603.28254 · HTML: https://mlsanity.com/paper/2603.28254/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 61 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [Momentum-Based Variance Reduction in Non-Convex SGD](https://mlsanity.com/paper/1905.10018/) (2019)
- [SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator](https://mlsanity.com/paper/1807.01695/) (2018)
- [Old Optimizer, New Norm: An Anthology](https://mlsanity.com/paper/2409.20325/) (2024)
- [SOAP: Improving and Stabilizing Shampoo using Adam](https://mlsanity.com/paper/2409.11321/) (2024)
- [Training Deep Learning Models with Norm-Constrained LMOs](https://mlsanity.com/paper/2502.07529/) (2025)
- [Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models](https://mlsanity.com/paper/2208.06677/) (2022)
- [The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm](https://mlsanity.com/paper/2505.16932/) (2025)
- [Convergence of Adam Under Relaxed Assumptions](https://mlsanity.com/paper/2304.13972/) (2023)
- [Muon Optimizes Under Spectral Norm Constraints](https://mlsanity.com/paper/2506.15054/) (2025)
- [NorMuon: Making Muon more efficient and scalable](https://mlsanity.com/paper/2510.05491/) (2025)
- [A Note on the Convergence of Muon](https://mlsanity.com/paper/2502.02900/) (2025)
- [Practical Efficiency of Muon for Pretraining](https://mlsanity.com/paper/2505.02222/) (2025)
- [Dion: Distributed Orthonormalized Updates](https://mlsanity.com/paper/2504.05295/) (2025)
- [AdaMuon: Adaptive Muon Optimizer](https://mlsanity.com/paper/2507.11005/) (2025)
- [MARS: Unleashing the Power of Variance Reduction for Training Large Models](https://mlsanity.com/paper/2411.10438/) (2024)
- [Stochastic Nested Variance Reduction for Nonconvex Optimization](https://mlsanity.com/paper/1806.07811/) (2018)
- [ASGO: Adaptive Structured Gradient Optimization](https://mlsanity.com/paper/2503.20762/) (2025)
- [Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures](https://mlsanity.com/paper/2311.00636/) (2023)
- [Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise](https://mlsanity.com/paper/2506.04192/) (2025)
- [SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients](https://mlsanity.com/paper/2106.08208/) (2021)
- [Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity](https://mlsanity.com/paper/2310.17998/) (2023)
- [Convergence of Muon with Newton-Schulz](https://mlsanity.com/paper/2601.19156/) (2026)

## Built on by (most cited first, 17 of 17)

- [Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers](https://mlsanity.com/paper/2605.18106/) (2026)
- [Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning](https://mlsanity.com/paper/2606.15576/) (2026)
- [Aurora: A Leverage-Aware Spectral Optimizer](https://mlsanity.com/paper/2606.27715/) (2026)
- [PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation](https://mlsanity.com/paper/2605.07067/) (2026)
- [Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction](https://mlsanity.com/paper/2606.01850/) (2026)
- [MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization](https://mlsanity.com/paper/2606.17526/) (2026)
- [Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization](https://mlsanity.com/paper/2606.27216/) (2026)
- [A Note on Stability for Orthogonalized Matrix Momentum with Client Sampling](https://mlsanity.com/paper/2606.01720/) (2026)
- [Muse: Representation Geometry of Muon Beyond Normalized Momentum](https://mlsanity.com/paper/2607.14536/) (2026)
- [MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning](https://mlsanity.com/paper/2608.05088/) (2026)
- [Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvature](https://mlsanity.com/paper/2608.09763/) (2026)
- [From SGD to Muon: Adaptive Optimization via Schatten-p Norms](https://mlsanity.com/paper/2605.19781/) (2026)
- [Blog: Survey of Optimizers](https://mlsanity.com/paper/2608.28557/) (2026)
- [Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training](https://mlsanity.com/paper/2609.36692/) (2026)
- [The Row Normalization Puzzle in Muon](https://mlsanity.com/paper/2609.39114/) (2026)
- [Optimizing Large Language Models with Chained LMOs](https://mlsanity.com/paper/2610.10975/) (2026)
- [From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon](https://mlsanity.com/paper/2610.11309/) (2026)

## Abstract (arXiv)

Orthogonalized-update optimizers such as Muon improve training of matrix-valued parameters, but existing extensions typically either rescale updates after orthogonalization or use heavier whitening-based preconditioners before it. We introduce method, a lightweight family of pre-orthogonalization equilibration schemes for Muon with three forms: two-sided row/column normalization (RC), row normalization (R), and column normalization (C). By rebalancing the momentum matrix before finite-step Newton--Schulz orthogonalization, method improves the geometry seen by orthogonalization. We show that finite-step orthogonalization is governed by the input spectrum, especially stable rank and condition number, and that row/column normalization acts as a zeroth-order surrogate for whitening. For hidden matrix weights, R is the default variant. Theoretically, method (R) retains the standard widetildemathcal O(T-1/4) Muon-type nonconvex stationarity guarantee with decoupled weight decay and a horizon-free diminishing learning-rate schedule, and extends it to finite-step NS5 up to an explicit inexactness constant. In LLaMA2 pretraining on C4, method (R) consistently outperforms Muon on 130M, 350M, and 1B models, with faster convergence and lower validation perplexity. The code is available at the hrefhttps://github.com/MaeChd/muon-eqMuonEq codebase.