Does Muon Need Fine-Grained Spectral Shaping?
Not on the map. Why?
Abstract (arXiv)
Muon combines current and past gradients into matrix momentum. For M=USigma V^top, the idealized polar update Q=UV^top gives every singular direction the same weight. We refer to this as the flat profile. Several recent optimizers replace this flat profile with fine-grained spectral maps that give each direction its own gain. We ask how much of this spectral detail a Muon update needs. Our spectral diagnostics show that approximately 94--97% of measured singular modes lie below an estimated noise edge, yet collectively align positively with a reference gradient. We introduce BulkBoost, a two-band spectral reweighting framework with fixed-rank and noise-calibrated variants. The latter uses split-minibatch gradient differences to calibrate a Marchenko--Pastur reference edge for Muon's Nesterov input, separating the bulk below the edge from the spikes above it. Both variants increase the bulk's relative weight through one shared gain while preserving the Frobenius norm of each matrix's unreweighted direction. For a fixed partition, our theory gives the first-order condition under which moving weight toward the bulk lowers the loss. It also quantifies the fraction of the maximal first-order improvement rate, over all per-mode reallocations, that two bands can capture. Across 30 continued-pretraining settings spanning Pythia-14M to 410M and six corpora, two-band reweighting is competitive with the fine-grained power-law profile of Freon and outperforms Spectra. Measured against Muon's flat profile, Freon reduces final loss by 0.022% of the pre-adaptation loss on average, whereas the two-band variants achieve reductions of 0.073--0.147%. These observations suggest that useful departures from the flat profile are surprisingly low-dimensional: a single bulk-to-spike gain captures at least as much benefit as the fine-grained spectral profiles.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 43 references that are in the map. First posted on arXiv 5 Oct 2026.
Builds on
21 of 43 references.
References · 21
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2017Decoupled Weight Decay Regularizationcited by 1,831
- 2025Muon is Scalable for LLM Trainingcited by 428
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
- 2018An Empirical Model of Large-Batch Trainingcited by 281
- 2024Old Optimizer, New Norm: An Anthologycited by 193
- 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
- 2025Dion: Distributed Orthonormalized Updatescited by 68
- 2025PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspectivecited by 52
- 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 14
- 2026Enhancing LLM Training via Spectral Clippingcited by 10
- 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 6
9 more
- 2026Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handlingcited by 4
- 2026Spectral Scaling Laws of Muoncited by 4
- 2026Dion3: Full-Stack Orthogonal Updatescited by 4
- 2026DynMuon: A Dynamic Spectral Shaping View of Muoncited by 3
- 2026The Spectral Dynamics and Noise Geometry of Muoncited by 3
- 2026Musec: MomentUm SpEctral Clipping for Stable Muon-type Trainingcited by 2
- 2026Muonp: Muon with Fractional Spectral Powerscited by 1
- 2026Optimizer Memory Schedules for Outscaling the Overtraining Axiscited by 1
- 2026Beyond the Matrix Sign: Quadratic Spectral Descentcited by 1
Built on by
Not cited yet.