mlsanityData to 7 Oct 2026

HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

arXiv 2603.10067 ↗First posted 10 Mar 2026cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon's orthogonalized update rule suppresses the emergence of heavy-tailed weight spectra and over-emphasizes the training along noise-dominated directions. Motivated by the Heavy-Tailed Self-Regularization (HT-SR) theory, we propose HTMuon. HTMuon preserves Muon's ability to capture parameter interdependencies while producing heavier-tailed updates and inducing heavier-tailed weight spectra. Experiments on LLM pretraining and image classification show that HTMuon consistently improves performance over state-of-the-art baselines and can also serve as a plug-in on top of existing Muon variants. For example, on LLaMA pretraining on the C4 dataset, HTMuon reduces perplexity by up to 0.98 compared to Muon. We further theoretically show that HTMuon corresponds to steepest descent under the Schatten-q norm constraint and provide convergence analysis in smooth non-convex settings. The implementation of HTMuon is available at https://github.com/TDCSZ327/HTmuon.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 56 of its 75 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

56 of 75 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  9. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  10. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  11. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  12. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
38 more
  1. 2025Muon is Scalable for LLM Trainingcited by 381
  2. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 303
  3. 20242 OLMo 2 Furiouscited by 302
  4. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  5. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  6. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  7. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  8. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 158
  9. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  10. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  11. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  12. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 87
  13. 2022When Do Flat Minima Optimizers Work?cited by 78
  14. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  15. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  16. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  17. 2024Modular Duality in Deep Learningcited by 71
  18. 2020Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networkscited by 62
  19. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  20. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  21. 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 53
  22. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
  23. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 38
  24. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  25. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  26. 2025On the Convergence of Muon and Beyondcited by 28
  27. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 28
  28. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  29. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 27
  30. 2025When do spectral gradient updates help in deep learning?cited by 25
  31. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 19
  32. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  33. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  34. 2024Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layerscited by 12
  35. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  36. 2024Model Balancing Helps Low-data Training and Fine-tuningcited by 11
  37. 2025Pretrained Image-Text Models are Secretly Video Captionerscited by 11
  38. 2025LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuningcited by 11

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 11
  2. 2026LionMuon: Alternating Spectral and Sign Descent for Efficient Trainingcited by 2
  3. 2026Convergence of Spectral Descent for Non-smooth Optimizationcited by 1
  4. 2026Muonp: Muon with Fractional Spectral Powerscited by 0
  5. 2026When to use what Schatten-p norm in deep learning?cited by 0
  6. 2026When Does Muon Help Agentic Reinforcement Learning?cited by 0
  7. 2026Beyond the Matrix Sign: Quadratic Spectral Descentcited by 0
  8. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0
  9. 2026Orthogonal Witness Control for Muon Optimization via Sigmoid Spectral Reshapingcited by 0
  10. 2026LionMuon: Alternating Spectral and Sign Descent for Efficient Trainingcited by 0
  11. 2026Muon meets Tamed Langevin: Momentum Preconditioning beyond Convex and gradient-Lipschitz Potentialscited by 0