mlsanityData to 7 Oct 2026

Training nGPT

arXiv 2608.01284 ↗First posted 2 Aug 2026cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

The normalized Transformer (nGPT) realizes hyperspherical representation learning by constraining model parameter vectors and activation vectors to the unit hypersphere. In this paper, we describe a practical training recipe for nGPT and evaluate it on modern hybrid Mamba-2--Transformer Mixture-of-Experts (MoE) models. The recipe introduces Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms. Compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 30B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens. The recipe scales across the models considered, which contain up to 30B total parameters.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 23 of its 34 references that are in the map. First posted on arXiv 2 Aug 2026.

Builds on

23 of 34 references.

References · 23

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  4. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  5. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  6. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  7. 2020Query-Key Normalization for Transformerscited by 266
  8. 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
  9. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  10. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  11. 2019Hyperspherical Prototype Networkscited by 85
  12. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
11 more
  1. 2018A Walk with SGDcited by 74
  2. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  3. 2018Spherical Latent Spaces for Stable Variational Autoencoderscited by 66
  4. 2017Deep Hyperspherical Learningcited by 55
  5. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  6. 2018Decoupled Networkscited by 45
  7. 2017Riemannian approach to batch normalizationcited by 35
  8. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 27
  9. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23
  10. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  11. 2023Weight Norm Controlcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Hyperparameter Transfer with Mixture-of-Expert Layers
  2. 2023From Sparse to Soft Mixtures of Experts
  3. 2026Normalized Architectures are Natively 4-Bit
  4. 2025Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training
  5. 2024Hexa-MoE: Efficient and Heterogeneous-aware Training for Mixture-of-Experts
  6. 2024Improving the Downstream Performance of Mixture-of-Experts Transformers via Weak Vanilla Transformers
  7. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate
  8. 2025Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems