Muon Meets Mamba: Spectral Optimization for State Space Models
On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language
Abstract (arXiv)
Muon is a recent optimizer that orthogonalizes the update to each weight matrix with a Newton-Schulz iteration, which performs steepest descent under the spectral norm. Almost all the evidence for it comes from Transformer models, and its behavior on state-space models is largely unreported. We compare Muon with AdamW on Mamba-2 130M under a controlled protocol that varies only which weight groups are trained with Muon. The benefit is localized. Muon on the output projection alone beats Muon on the input projection or on both. The advantage is mainly one of token efficiency. It holds on two corpora and two token budgets, and persists when training continues well past the compute-optimal point. Conditioning does not explain the gain. Muon lowers the condition number of whichever projection it trains, but the better-conditioned input projection is not the one that helps.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 16 of its 22 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
16 of 22 references.
Closest ancestors · 6
Same direction, newest first
- 2024An Empirical Study of Mamba-based Language Models12 Jun
- 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
- 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
- 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks6 Feb
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
- 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct
Other references · 10
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2025Muon is Scalable for LLM Trainingcited by 428
- 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
- 2024Old Optimizer, New Norm: An Anthologycited by 193
- 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
- 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 80
- 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
- 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Muon: Training and Trade-offs with Latent Attention and MoE
- 2025MuLoCo: Muon is a practical inner optimizer for DiLoCo
- 2026An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies
- 2026AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
- 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR
- 2026MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
- 2026Muon2: Boosting Muon via Adaptive Second-Moment Preconditioning
- 2026Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining