mlsanityData to 14 Oct 2026

Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization

arXiv 2610.02798 ↗First posted 2 Oct 2026cs.LG math.OC stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training, yet the mechanisms of this transformation in feature learning remain poorly understood. In this work, we investigate this question through a tractable factual-recall model, where a fact maps each subject-relation pair to an answer, and a linear transformer learns the subject- and relation-dependent information required to recover this mapping. The transformer is optimized with gradient flow (GF), spectral GF, or Sign GF, which are continuous-time limits of gradient descent, Muon, and Adam, respectively. Prior studies (Nichani et al., 2025) have shown that when the number of subjects exceeds the number of relations, GF learns relation-dependent information before subject-dependent information, producing a feature-separation phase during training. We characterize this separation with the learning times when the subject- and relation-dependent components of the prediction reach a target accuracy. With S subjects and R relations, GF has a learning-time ratio of widetildeTheta(√S/R), whereas Spectral GF reduces this ratio to widetildeTheta(1). In addition, for fixed S and R, the subject- and relation-dependent errors decay as 1/(Tlog T) in training time T under GF, but as exp(-poly(T)) under spectral GF. Finally, we show that GF and spectral GF are equivariant under orthogonal transformations of the token embeddings, whereas Sign GF is not: Different orthonormal embeddings can potentially produce no feature separation, a large feature-separation phase, or even a reversed learning order. These results provide a mechanistic view of how spectral orthogonalization can fundamentally reshape feature-learning dynamics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

32 of 39 references.

References · 32

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  3. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  4. 2022Transformers learn in-context by gradient descentcited by 732
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  6. 2025Kimi K2: Open Agentic Intelligencecited by 567
  7. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  8. 2025Muon is Scalable for LLM Trainingcited by 428
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  10. 2024Old Optimizer, New Norm: An Anthologycited by 193
  11. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  12. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
20 more
  1. 2018Risk and parameter convergence of logistic regressioncited by 85
  2. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  3. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 51
  4. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  5. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  6. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  7. 2023Scaling Laws for Associative Memoriescited by 35
  8. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
  9. 2023On the Implicit Bias of Adamcited by 27
  10. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  11. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 19
  12. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 18
  13. 2024Learning Associative Memories with Gradient Descentcited by 16
  14. 2026Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memorycited by 14
  15. 2026Muon in Associative Memory Learning: Training Dynamics and Scaling Lawscited by 13
  16. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 11
  17. 2026To Use or not to Use Muon: How Simplicity Bias in Optimizers Matterscited by 10
  18. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 6
  19. 2026Muon Learns More Robust and Transferable Features than Adamcited by 3
  20. 2026Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamicscited by 3

Built on by

Not cited yet.