mlsanityData to 7 Oct 2026

Muse: Representation Geometry of Muon Beyond Normalized Momentum

arXiv 2607.14536 ↗First posted 16 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Muon-style optimizers apply a polar map to matrix momentum, but their updates also depend on the representation of each parameter block before orthogonalization. We study this representation choice as a form of optimizer geometry and introduce method, a family of Muon-style optimizers that shares the same momentum rule and Newton--Schulz backend across native, nearest-square, skinny, and vector representations. Each Frobenius-isometric representation induces a distinct polar steepest-descent geometry, in which the shorter matrix dimension determines the number of supported singular channels, the pullback scaling, and the constants in stochastic nonconvex convergence bounds. In a teacher--student model, curvature collapse and an isotropic Marchenko--Pastur spectral profile connect early-stage dissipation to the represented nuclear-to-squared-Frobenius norm ratio. Pretraining experiments on LLaMA2-130M and LLaMA2-600M, together with fixed-momentum diagnostics, show that balanced non-native representations can match the performance of the native representation, whereas reducing the shorter dimension weakens the scaling and singular-channel support, leading to behavior that increasingly resembles normalized momentum.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 59 of its 75 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

59 of 75 references.

References · 59

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  5. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  6. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  7. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  8. 2025Muon is Scalable for LLM Trainingcited by 381
  9. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  10. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 283
  11. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  12. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
38 more
  1. 2024Old Optimizer, New Norm: An Anthologycited by 173
  2. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  3. 2020Momentum Improves Normalized SGDcited by 136
  4. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  5. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  6. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  7. 2025A Note on the Convergence of Muoncited by 78
  8. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  9. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  10. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  11. 2025Dion: Distributed Orthonormalized Updatescited by 64
  12. 2018Stochastic Nested Variance Reduction for Nonconvex Optimizationcited by 63
  13. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 59
  14. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  15. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  16. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  17. 2025PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspectivecited by 48
  18. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  19. 2023Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexitycited by 38
  20. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  21. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 34
  22. 2026Convergence of Muon with Newton-Schulzcited by 30
  23. 2025On the Convergence of Muon and Beyondcited by 28
  24. 2025Convergence Bound and Critical Batch Size of Muon Optimizercited by 24
  25. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 23
  26. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 22
  27. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  28. 2023Sketchy: Memory-efficient Adaptive Regularization with Frequent Directionscited by 16
  29. 2026MuonEq: Balancing Before Orthogonalization with Lightweight Equilibrationcited by 16
  30. 2025A Stable Whitening Optimizer for Efficient Neural Network Trainingcited by 15
  31. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  32. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 13
  33. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  34. 2025Error Feedback for Muon and Friendscited by 11
  35. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 11
  36. 2026RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimizationcited by 11
  37. 2025Effective Quantization of Muon Optimizer Statescited by 10
  38. 2020Stochastic Normalized Gradient Descent with Momentum for Large-Batch Trainingcited by 9

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.