Learning Rate Transfer in Normalized Transformers
Not on the map. Why?
Abstract (arXiv)
The Normalized Transformer, or nGPT (arXiv:2410.01131) achieves impressive training speedups and does not require weight decay or learning rate warmup. However, despite having hyperparameters that explicitly scale with model size, we observe that nGPT does not exhibit learning rate transfer across model dimension and token horizon. To rectify this, we combine numerical experiments with a principled use of alignment exponents (arXiv:2407.05872) to revisit and modify the μP approach to hyperparameter transfer (arXiv:2011.14522). The result is a novel nGPT parameterization we call nuGPT. Through extensive empirical validation, we find nuGPT exhibits learning rate transfer across width, depth, and token horizon.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 29 Apr 2026.
Builds on
33 of 41 references.
References · 33
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
- 20242 OLMo 2 Furiouscited by 340
- 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
- 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
- 2023A Spectral Condition for Feature Learningcited by 109
- 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
21 more
- 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
- 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 100
- 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
- 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
- 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
- 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76
- 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
- 2024Scalable Optimization in the Modular Normcited by 68
- 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
- 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
- 2020Tensor Programs III: Neural Matrix Lawscited by 49
- 2024Scaling Optimal LR Across Token Horizonscited by 40
- 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
- 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 39
- 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
- 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 27
- 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
- 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
- 2026Rethinking Language Model Scaling under Transferable Hypersphere Optimizationcited by 12
- 2024An Empirical Study of μP Learning Rate Transfercited by 10
- 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 10
Built on by
Citing papers per year: 2026: 3.
3 papers.