mlsanityData to 7 Oct 2026

AdaMuon: Adaptive Muon Optimizer

arXiv 2507.11005 ↗First posted 15 Jul 2025cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

We propose AdaMuon, a novel optimizer that combines element-wise adaptivity with orthogonal updates for large-scale neural network training. AdaMuon incorporates two tightly coupled mechanisms: (1) an element-wise second momentum estimator applied to orthogonalized update directions, and (2) a sign-stabilized orthogonal update, where the momentum is first sign-transformed before orthogonalization. These two components jointly enable variance-adaptive scaling while maintaining stable update geometry. In addition, AdaMuon employs an RMS-aligned rescaling strategy to match the root-mean-square update magnitude to Adam, allowing direct reuse of existing learning rate schedules without extra tuning. Experiments demonstrate that AdaMuon not only maintains stability but can surpass Adam by more than 40% training efficiency in large-scale scenarios.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 15 Jul 2025.

Builds on

28 of 40 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
16 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  5. 2019On the Convergence of Adam and Beyondcited by 834
  6. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  8. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  9. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  10. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  11. 2025Muon is Scalable for LLM Trainingcited by 381
  12. 2024Old Optimizer, New Norm: An Anthologycited by 173
  13. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  14. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  15. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65
  16. 2025Muon Optimizer Accelerates Grokkingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 44.

59 papers.

Sort:

By area:

  1. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  2. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
  3. 2025On the Convergence of Muon and Beyondcited by 28
  4. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 22
  5. 2025GeoTransolver: Learning Physics on Irregular Domains Using Multi-scale Geometry Aware Physics Attention Transformercited by 20
  6. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 17
  7. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 16
  8. 2025What Really Matters in Matrix-Whitening Optimizers?cited by 15
  9. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 15
  10. 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 15
  11. 2026Muon2: Boosting Muon via Adaptive Second-Moment Preconditioningcited by 15
  12. 2025ROOT: Robust Orthogonalized Optimizer for Neural Network Trainingcited by 13
38 more
  1. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 13
  2. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 11
  3. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 11
  4. 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 8
  5. 2025MuonAll: Muon Variant for Efficient Finetuning of Large Language Modelscited by 6
  6. 2026Stabilizing Native Low-Rank LLM Pretrainingcited by 6
  7. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5
  8. 2026MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-trainingcited by 5
  9. 2026Enhancing LLM Training via Spectral Clippingcited by 5
  10. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 5
  11. 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 5
  12. 2025DeMuon: A Decentralized Muon for Matrix Optimization over Graphscited by 4
  13. 2025Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimizationcited by 4
  14. 2026The Agentic Researcher: A Practical Guide to AI-Assisted Research in Mathematics and Machine Learningcited by 4
  15. 2026Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Trainingcited by 4
  16. 2026Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergencecited by 4
  17. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 4
  18. 2025Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Trainingcited by 3
  19. 2026OLion: Approaching the Hadamard Ideal by Intersecting Spectral and ℓ∞ Implicit Biasescited by 3
  20. 2026Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimizationcited by 3
  21. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  22. 2026A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muoncited by 2
  23. 2026When and Why SignSGD Outperforms SGD: A Theoretical Study Based on ℓ1-norm Lower Boundscited by 2
  24. 2026PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisationcited by 2
  25. 2025Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rulescited by 1
  26. 2026Convergence of Spectral Descent for Non-smooth Optimizationcited by 1
  27. 2026A Note on Stability for Orthogonalized Matrix Momentum with Client Samplingcited by 1
  28. 2026OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimalitycited by 1
  29. 2026Muon Learns More Robust and Transferable Features than Adamcited by 1
  30. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 1
  31. 2026MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioningcited by 1
  32. 2026Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muoncited by 1
  33. 2026Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretrainingcited by 0
  34. 2026From SGD to Muon: Adaptive Optimization via Schatten-p Normscited by 0
  35. 2026MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Trainingcited by 0
  36. 2026Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?cited by 0
  37. 2026Stochastic convergence of parallel asynchronous adaptive first-order methodscited by 0
  38. 2026FOGO: Forgetting-aware Orthogonalization Optimizercited by 0

50 most cited of 59 All 59 (JSON)