mlsanityData to 7 Oct 2026

nGPT: Normalized Transformer with Representation Learning on the Hypersphere

arXiv 2410.01131 ↗First posted 1 Oct 2024cs.LG cs.AIcited by 68

Not on the map. Why?

Abstract (arXiv)

We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP, attention matrices and hidden states are unit norm normalized. The input stream of tokens travels on the surface of a hypersphere, with each layer contributing a displacement towards the target output predictions. These displacements are defined by the MLP and attention blocks, whose vector components also reside on the same hypersphere. Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

27 of 47 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  6. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  7. 2020On Layer Normalization in the Transformer Architecturecited by 906
  8. 2022Transformers learn in-context by gradient descentcited by 714
  9. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  10. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  11. 2020Query-Key Normalization for Transformerscited by 266
  12. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
15 more
  1. 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
  2. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  3. 2019Hyperspherical Prototype Networkscited by 85
  4. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  5. 2018Spherical Latent Spaces for Stable Variational Autoencoderscited by 66
  6. 2024Scalable Optimization in the Modular Normcited by 64
  7. 2017Deep Hyperspherical Learningcited by 55
  8. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  9. 2018Decoupled Networkscited by 45
  10. 2018Fix your classifier: the marginal value of training the last weight layercited by 43
  11. 2024Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Modelscited by 38
  12. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23
  13. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  14. 2018Adaptive regularization with cubics on manifoldscited by 17
  15. 2023Weight Norm Controlcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 27; 2026: 38.

68 papers.

Sort:

By area:

  1. 2025Transformers without Normalizationcited by 69
  2. 2025Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and ASTcited by 51
  3. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  4. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 32
  5. 2025Superposition Yields Robust Neural Scalingcited by 30
  6. 2025The Mean-Field Dynamics of Transformerscited by 24
  7. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  8. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  9. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  10. 2025Cautious Weight Decaycited by 12
  11. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 12
  12. 2025Normalization in Attention Dynamicscited by 9
38 more
  1. 2025CODA: Repurposing Continuous VAEs for Discrete Tokenizationcited by 8
  2. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 8
  3. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 8
  4. 2025Spherical Leech Quantization for Visual Tokenization and Generationcited by 6
  5. 2025Learning in Compact Spaces with Approximately Normalized Transformercited by 5
  6. 2025Optimal Scaling Needs Optimal Normcited by 5
  7. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  8. 2024Pyramid Vector Quantization for LLMscited by 4
  9. 2026The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topologycited by 4
  10. 2024Interpreting Neural Networks through Mahalanobis Distancecited by 3
  11. 2026NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networkscited by 3
  12. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  13. 2025Hyper-SET: Designing Transformers via Hyperspherical Energy Minimizationcited by 2
  14. 2025ENSAM: an efficient foundation model for interactive segmentation of 3D medical imagescited by 2
  15. 2025Zero-Shot Performance Prediction for Probabilistic Scaling Lawscited by 2
  16. 2026SpanNorm: Reconciling Training Stability and Performance in Deep Transformerscited by 2
  17. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 2
  18. 2026SimpleGPT: Improving GPT via A Simple Normalization Strategycited by 2
  19. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  20. 2024A Mechanistic Study of Transformers Training Dynamicscited by 1
  21. 2025Normalized Matching Transformercited by 1
  22. 2025DNT: a Deeply Normalized Transformer that can be trained by Momentum SGDcited by 1
  23. 2025KEEP: Integrating Medical Ontologies with Clinical Data for Robust Code Embeddingscited by 1
  24. 2025Lite ENSAM: a lightweight cancer segmentation model for 3D Computed Tomographycited by 1
  25. 2026GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimizationcited by 1
  26. 2026Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Expertscited by 1
  27. 2026Reward-Conditioned Reinforcement Learningcited by 1
  28. 2026Polaris: Coupled Orbital Polar Embeddings for Hierarchical Concept Learningcited by 1
  29. 2026Demystifying Manifold Constraints in LLM Pre-trainingcited by 1
  30. 2026Normalized Architectures are Natively 4-Bitcited by 1
  31. 2026Kuramoto Attention: Synchronizing Self-Attention on the Toruscited by 1
  32. 2026Beyond a Single Explanation of the Adam--SGD Gapcited by 1
  33. 2026Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Modelscited by 1
  34. 2025Motif 2.6B Technical Reportcited by 0
  35. 2025Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?cited by 0
  36. 2025Sparse Concept Anchoring for Interpretable and Controllable Neural Representationscited by 0
  37. 2025Bright 4B: Scaling Hyperspherical Learning for Segmentation in 3D Brightfield Microscopycited by 0
  38. 2026Torus embeddingscited by 0

50 most cited of 68 All 68 (JSON)