mlsanityData to 14 Oct 2026

Practical Efficiency of Muon for Pretraining

arXiv 2505.02222 ↗First posted 4 May 2025cs.LG stat.MLcited by 80

Not on the map. Why?

Abstract (arXiv)

We demonstrate that Muon, the simplest instantiation of a second-order optimizer, explicitly expands the Pareto frontier over AdamW on the compute-time tradeoff. We find that Muon is more effective than AdamW in retaining data efficiency at large batch sizes, far beyond the so-called critical batch size, while remaining computationally efficient, thus enabling more economical training. We study the combination of Muon and the maximal update parameterization (muP) for efficient hyperparameter transfer and present a simple telescoping algorithm that accounts for all sources of error in muP while introducing only a modest overhead in resources. We validate our findings through extensive experiments with model sizes up to four billion parameters and ablations on the data distribution and architecture.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 4 May 2025.

Builds on

34 of 49 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  4. 2021Scaling Vision Transformerscited by 999
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  6. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  7. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  8. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  9. 2025Muon is Scalable for LLM Trainingcited by 428
  10. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  11. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  12. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
22 more
  1. 2018An Empirical Model of Large-Batch Trainingcited by 281
  2. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  3. 2024Old Optimizer, New Norm: An Anthologycited by 193
  4. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  5. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  6. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  7. 2023A Spectral Condition for Feature Learningcited by 109
  8. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76
  9. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  10. 2024Scalable Optimization in the Modular Normcited by 68
  11. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  12. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43
  13. 2016Coupling Adaptive Batch Sizes with Learning Ratescited by 30
  14. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  15. 2023Effective Theory of Transformers at Initializationcited by 21
  16. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  17. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 19
  18. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  19. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 15
  20. 2024μP2: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scalingcited by 10
  21. 2023Tune As You Scale: Hyperparameter Optimization For Compute Efficient Trainingcited by 9
  22. 2024TASI Lectures on Physics for Machine Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 51.

80 papers.

Sort:

By area:

  1. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 110
  2. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  3. 2025AdaMuon: Adaptive Muon Optimizercited by 67
  4. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  5. 2025In Search of Adam's Secret Saucecited by 40
  6. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 37
  7. 2026Convergence of Muon with Newton-Schulzcited by 36
  8. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 35
  9. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 35
  10. 2025On the Convergence of Muon and Beyondcited by 31
  11. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 30
  12. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 28
38 more
  1. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  2. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23
  3. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
  4. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 18
  5. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 18
  6. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 17
  7. 2026Muon2: Boosting Muon via Adaptive Second-Moment Preconditioningcited by 17
  8. 2025FedMuon: Accelerating Federated Learning with Matrix Orthogonalizationcited by 16
  9. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
  10. 2025The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvementscited by 15
  11. 2025Dion2: A Simple Method to Shrink Matrix in Muoncited by 13
  12. 2026Insights on Muon from Simple Quadraticscited by 13
  13. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  14. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 12
  15. 2025Effective Quantization of Muon Optimizer Statescited by 11
  16. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 11
  17. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 9
  18. 2025Optimal Scaling Needs Optimal Normcited by 8
  19. 2025MuonAll: Muon Variant for Efficient Finetuning of Large Language Modelscited by 7
  20. 2026Can Muon Fine-tune Adam-Pretrained Models?cited by 7
  21. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
  22. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 6
  23. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5
  24. 2026Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergencecited by 5
  25. 2025Non-Euclidean Broximal Point Method: A Blueprint for Geometry-Aware Optimizationcited by 4
  26. 2025Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimizationcited by 4
  27. 2026PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Trainingcited by 4
  28. 2025Muon: Training and Trade-offs with Latent Attention and MoEcited by 3
  29. 2026NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networkscited by 3
  30. 2026Phases of Muon: When Muon Eclipses SignSGDcited by 3
  31. 2026The Spectral Dynamics and Noise Geometry of Muoncited by 3
  32. 2026Muon Learns More Robust and Transferable Features than Adamcited by 3
  33. 2025Federated Stochastic Minimax Optimization under Heavy-Tailed Noisescited by 2
  34. 2026MuonRec: Shifting the Optimizer Paradigm Beyond Adam in Scalable Generative Recommendationcited by 2
  35. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2
  36. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  37. 2026Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Methodcited by 2
  38. 2026Convergence of Spectral Descent for Non-smooth Optimizationcited by 2

50 most cited of 80 All 80 (JSON)