A Proof of Learning Rate Transfer under μP
Not on the map. Why?
Abstract (arXiv)
We provide the first proof of learning rate transfer with width in a linear multi-layer perceptron (MLP) parametrized with μP, a neural network parameterization designed to ``maximize'' feature learning in the infinite-width limit. We show that under μ P, the optimal learning rate converges to a non-zero constant as width goes to infinity, providing a theoretical explanation to learning rate transfer. In contrast, we show that this property fails to hold under alternative parametrizations such as Standard Parametrization (SP) and Neural Tangent Parametrization (NTP). We provide intuitive proofs and support the theoretical findings with extensive empirical results.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 3 Nov 2025.
Builds on
32 of 45 references.
References · 32
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
- 2016Deep Information Propagationcited by 194
- 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
- 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
20 more
- 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
- 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
- 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 94
- 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
- 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
- 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
- 2025Dion: Distributed Orthonormalized Updatescited by 64
- 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
- 2020Stable ResNetcited by 49
- 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
- 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
- 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
- 2022Infinite-width limit of deep linear neural networkscited by 20
- 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
- 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 17
- 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
- 2024An Empirical Study of μP Learning Rate Transfercited by 9
- 2025Scaling Diffusion Transformers Efficiently via μPcited by 6
- 2019Exact Convergence Rates of the Neural Tangent Kernel in the Large Depth Limitcited by 5
- 2025Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Sizecited by 4
Built on by
Citing papers per year: 2026: 3.
3 papers.