mlsanityData to 7 Oct 2026

Muon is Scalable for LLM Training

arXiv 2502.16982 ↗First posted 24 Feb 2025cs.LG cs.AI cs.CLcited by 381

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves ~!2× computational efficiency compared to AdamW with compute optimal training. Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research.

Key facts. Cited by 381 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 34 of its 46 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

34 of 46 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
22 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  10. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  11. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  12. 20242 OLMo 2 Furiouscited by 302
  13. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  14. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  15. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  16. 2024Old Optimizer, New Norm: An Anthologycited by 173
  17. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 89
  18. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  19. 2015Preconditioned Stochastic Gradient Descentcited by 48
  20. 2024Curvature-Informed SGD via General Purpose Lie-Group Preconditionerscited by 16
  21. 2018Preconditioner on Matrix Lie Group for SGDcited by 15
  22. 2022Black Box Lie Group Preconditioners for SGDcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 98; 2026: 283.

381 papers.

Sort:

By area:

  1. 2025Kimi K2: Open Agentic Intelligencecited by 523
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  3. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  4. 2025Kimi-VL Technical Reportcited by 381
  5. 2025Kimi-Audio Technical Reportcited by 210
  6. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  7. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  8. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  9. 2026Kimi K3: Open Frontier Intelligencecited by 99
  10. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  11. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  12. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
38 more
  1. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  2. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  3. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 66
  4. 2025Dion: Distributed Orthonormalized Updatescited by 64
  5. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  6. 2026Attention Residualscited by 57
  7. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  8. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  9. 2026Ultralytics YOLO26: Unified Real-Time End-to-End Vision Modelscited by 47
  10. 2025In Search of Adam's Secret Saucecited by 40
  11. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  12. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
  13. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 34
  14. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 30
  15. 2026Convergence of Muon with Newton-Schulzcited by 30
  16. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  17. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  18. 2025On the Convergence of Muon and Beyondcited by 28
  19. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 28
  20. 2025An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variantscited by 25
  21. 2026OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoningcited by 25
  22. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 24
  23. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 23
  24. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 22
  25. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  26. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  27. 2026Toto 2.0: Time Series Forecasting Enters the Scaling Eracited by 21
  28. 2025LiMuon: Light and Fast Muon Optimizer for Large Modelscited by 20
  29. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  30. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 17
  31. 2025Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidencecited by 17
  32. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 16
  33. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 16
  34. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 16
  35. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 16
  36. 2025FedMuon: Accelerating Federated Learning with Matrix Orthogonalizationcited by 15
  37. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 15
  38. 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 15

50 most cited of 381 All 381 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Muon: Training and Trade-offs with Latent Attention and MoE
  2. 2026FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
  3. 2026How Much Orthogonalization Does Muon Need?
  4. 2026Move on Muon : A Hamiltonian probability gradient flow perspective of Muon optimizer
  5. 2026AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
  6. 2026Muon-C: Operator-Aligned Muon for Convolutional Kernels
  7. 2026Rethinking Language Model Scaling under Transferable Hypersphere Optimization
  8. 2026Controlled LLM Training on Spectral Sphere