mlsanityData to 7 Oct 2026

Controlled LLM Training on Spectral Sphere

arXiv 2601.08393 ↗First posted 13 Jan 2026cs.LG cs.AIcited by 28

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Scaling large models requires optimization strategies that ensure rapid convergence grounded in stability. Maximal Update Parametrization (boldsymbolμP) provides a theoretical safeguard for width-invariant Theta(1) activation control, whereas emerging optimizers like Muon are only ``half-aligned'' with these constraints: they control updates but allow weights to drift. To address this limitation, we introduce the Spectral Sphere Optimizer (SSO), which enforces strict module-wise spectral constraints on both weights and their updates. By deriving the steepest descent direction on the spectral sphere, SSO realizes a fully boldsymbolμP-aligned optimization process. To enable large-scale training, we implement SSO as an efficient parallel algorithm within Megatron. Through extensive pretraining on diverse architectures, including Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet models, SSO consistently outperforms AdamW and Muon. Furthermore, we observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 19 of its 33 references that are in the map. First posted on arXiv 13 Jan 2026.

Builds on

19 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models21 Jan

Other references · 18

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  9. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  10. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  11. 2020Query-Key Normalization for Transformerscited by 266
  12. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
6 more
  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  2. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  3. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  4. 2023A Spectral Condition for Feature Learningcited by 101
  5. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  6. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 28.

28 papers.

Sort:

By area:

  1. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 23
  2. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 21
  3. 2026Spherical Steering: Geometry-Aware Activation Rotation for Language Modelscited by 17
  4. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  5. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 12
  6. 2026Rethinking Language Model Scaling under Transferable Hypersphere Optimizationcited by 11
  7. 2026Manifold constrained steepest descent for smooth and closed-set optimizationcited by 8
  8. 2026Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifoldscited by 6
  9. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 5
  10. 2026Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizerscited by 4
  11. 2026Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizercited by 4
  12. 2026Orthogonal Model Mergingcited by 3
16 more
  1. 2026POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformationcited by 3
  2. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 3
  3. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  4. 2026Hyperball May Not Be a Free Lunchcited by 3
  5. 2026Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformationcited by 2
  6. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  7. 2026Effective Learning Rate Governs Loss Dynamics in Language Model Pretrainingcited by 2
  8. 2026Conditional Optimal Bridge for Riemannian Activation Steeringcited by 1
  9. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 1
  10. 2026Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectivescited by 0
  11. 2026Rethinking Bregman Divergences in Kronecker-Factored Optimizerscited by 0
  12. 2026Redesign Mixture-of-Experts Routers with Manifold Power Iterationcited by 0
  13. 2026Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretrainingcited by 0
  14. 2026HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperballcited by 0
  15. 2026Quantization-Aware Pre-Training with Constrained Empirical Weight Distributioncited by 0
  16. 2026Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training
  2. 2026Orthogonal Witness Control for Muon Optimization via Sigmoid Spectral Reshaping
  3. 2026Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimization
  4. 2026Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
  5. 2026An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies
  6. 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
  7. 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR
  8. 2026Muon Meets Mamba: Spectral Optimization for State Space Models