mlsanityData to 14 Oct 2026

Muonp: Muon with Fractional Spectral Powers

arXiv 2606.13867 ↗First posted 11 Jun 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Muon is an increasingly widely used optimizer that replaces a gradient G=USV^top with its polar factor UV^top, thereby flattening the singular spectrum. However, full flattening discards singular-value information that may matter for adaptation. We introduce Muonp, a Muon-style optimizer that instead uses fractional spectral-power updates USpV^top for rational p∈(0,1), interpolating between Muon and gradient descent. To make it practical, we prove that fractional spectral powers cannot be computed by any fixed univariate polynomial iteration, and furthermore derive low-degree odd bivariate recurrences that approximate USpV^top using only matrix multiplications, preserving Muon's matrix-multiplication-only structure and compute complexity. We show that Muonp maximizes the linear improvement in loss under the Schatten q-norm for q=1+frac1p. Empirically, Muonp is especially effective for finetuning: on billion-scale models, Muonp improves validation perplexity and downstream task performance. We further analyze when Muonp is less suitable, through the lens of spectral geometry. Our results reveal important insights on when preserving the singular spectrum can bring significant gains, and introduce a principled way to achieve them.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

33 of 46 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  5. 2025Muon is Scalable for LLM Trainingcited by 428
  6. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  7. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  8. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  9. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  10. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 110
  11. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  12. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
21 more
  1. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  2. 2025NorMuon: Making Muon more efficient and scalablecited by 84
  3. 2025A Note on the Convergence of Muoncited by 83
  4. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  5. 2025AdaMuon: Adaptive Muon Optimizercited by 67
  6. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 60
  7. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 58
  8. 2025PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspectivecited by 52
  9. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  10. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  11. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 19
  12. 2025ROOT: Robust Orthogonalized Optimizer for Neural Network Trainingcited by 15
  13. 2026Muon in Associative Memory Learning: Training Dynamics and Scaling Lawscited by 13
  14. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 12
  15. 2026Can Muon Fine-tune Adam-Pretrained Models?cited by 7
  16. 2025PiCa: Parameter-Efficient Fine-Tuning with Column Space Projectioncited by 6
  17. 2025Weight Spectra Induced Efficient Model Adaptationcited by 5
  18. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 5
  19. 2025POME: Post Optimization Model Edit via Muon-style Projectioncited by 4
  20. 2026Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Trainingcited by 4
  21. 2026IFNSO: Iteration-Free Newton-Schulz Orthogonalizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Does Muon Need Fine-Grained Spectral Shaping?cited by 0