mlsanityData to 14 Oct 2026

Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit

arXiv 2309.16620 ↗First posted 28 Sep 2023stat.ML cond-mat.dis-nn cs.AIcited by 66

Not on the map. Why?

Abstract (arXiv)

The cost of hyperparameter tuning in deep learning has been rising with model sizes, prompting practitioners to find new tuning methods using a proxy of smaller networks. One such proposal uses μP parameterized networks, where the optimal hyperparameters for small width networks transfer to networks with arbitrarily large width. However, in this scheme, hyperparameters do not transfer across depths. As a remedy, we study residual networks with a residual branch scale of 1/√depth in combination with the μP parameterization. We provide experiments demonstrating that residual architectures including convolutional ResNets and Vision Transformers trained with this parameterization exhibit transfer of optimal hyperparameters across width and depth on CIFAR-10 and ImageNet. Furthermore, our empirical findings are supported and motivated by theory. Using recent developments in the dynamical mean field theory (DMFT) description of neural network learning dynamics, we show that this parameterization of ResNets admits a well-defined feature learning joint infinite-width and infinite-depth limit and show convergence of finite-size network dynamics towards this limit.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 54 of its 68 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

54 of 68 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2021Scaling Vision Transformerscited by 999
  11. 2020On Layer Normalization in the Transformer Architecturecited by 922
  12. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
38 more
  1. 2018On Lazy Training in Differentiable Programmingcited by 675
  2. 2017Deep Neural Networks as Gaussian Processescited by 622
  3. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  4. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  5. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  7. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  8. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  9. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  10. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  11. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  12. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  13. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  14. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  15. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  16. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  17. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  18. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  19. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  20. 2020Stable ResNetcited by 49
  21. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  22. 2020On the distance between two neural networks and the stability of learningcited by 46
  23. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 43
  24. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  25. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 41
  26. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  27. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 29
  28. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  29. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  30. 2018Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Functioncited by 22
  31. 2019Finite size corrections for neural network Gaussian processescited by 22
  32. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  33. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  34. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  35. 2021Unified field theoretical approach to deep and recurrent neuronal networkscited by 20
  36. 2022The Influence of Learning Rule on Representation Dynamics in Wide Neural Networkscited by 19
  37. 2022Scaling Laws For Deep Learning Based Image Reconstructioncited by 14
  38. 2023Automatic Gradient Descent: Deep Learning without Hyperparameterscited by 14

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 16; 2025: 24; 2026: 25.

66 papers.

Sort:

By area:

  1. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  2. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  3. 2024Scaling Laws for Precisioncited by 95
  4. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  5. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  6. 2024Scalable Optimization in the Modular Normcited by 68
  7. 2025The Curse of Depth in Large Language Modelscited by 62
  8. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  9. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
  10. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43
  11. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
  12. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretrainingcited by 34
38 more
  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  2. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 24
  3. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  4. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 23
  5. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  6. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
  7. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  8. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 19
  9. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 17
  10. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  11. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  12. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 16
  13. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
  14. 2024High-dimensional learning of narrow neural networkscited by 12
  15. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  16. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  17. 2026On the Residual Scaling of Looped Transformers: Stability and Transferabilitycited by 10
  18. 2024Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limitcited by 9
  19. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
  20. 2026Scaling View Synthesis Transformerscited by 9
  21. 2025Optimal Scaling Needs Optimal Normcited by 8
  22. 2024Predictive Coding Networks and Inference Learning: Tutorial and Surveycited by 7
  23. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 7
  24. 2026Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Modelcited by 7
  25. 2026DeepLoop: Depth Scaling for Looped Transformerscited by 7
  26. 2025Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Sizecited by 6
  27. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  28. 2026Scaling laws for amplitude surrogatescited by 6
  29. 2025Function-Space Learning Ratescited by 4
  30. 2025Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μP Parametrizationcited by 4
  31. 2025Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexitycited by 4
  32. 2025A Proof of Learning Rate Transfer under μPcited by 3
  33. 2026Learning Rate Transfer in Normalized Transformerscited by 3
  34. 2026On the Infinite Width and Depth Limits of Predictive Coding Networkscited by 2
  35. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  36. 2026Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Ratecited by 2
  37. 2026Statistical Properties of Training & Generalizationcited by 2
  38. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2

50 most cited of 66 All 66 (JSON)