mlsanityData to 7 Oct 2026

Meta-Principled Family of Hyperparameter Scaling Strategies

arXiv 2210.04909 ↗First posted 10 Oct 2022cs.LG hep-th stat.MLcited by 18

Not on the map. Why?

Abstract (arXiv)

In this note, we first derive a one-parameter family of hyperparameter scaling strategies that interpolates between the neural-tangent scaling and mean-field/maximal-update scaling. We then calculate the scalings of dynamical observables -- network outputs, neural tangent kernels, and differentials of neural tangent kernels -- for wide and deep neural networks. These calculations in turn reveal a proper way to scale depth with width such that resultant large-scale models maintain their representation-learning ability. Finally, we observe that various infinite-width limits examined in the literature correspond to the distinct corners of the interconnected web spanned by effective theories for finite-width neural networks, with their training dynamics ranging from being weakly-coupled to being strongly-coupled.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

30 of 43 references.

References · 30

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2022A ConvNet for the 2020scited by 3,545
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  11. 2018On Lazy Training in Differentiable Programmingcited by 671
  12. 2017Deep Neural Networks as Gaussian Processescited by 617
18 more
  1. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  2. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  3. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  4. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  5. 2016On the Expressive Power of Deep Neural Networkscited by 302
  6. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  7. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  8. 2016Deep Information Propagationcited by 194
  9. 2021The Principles of Deep Learning Theorycited by 169
  10. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  11. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  12. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  13. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  14. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  15. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  16. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  17. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  18. 2020On the asymptotics of wide networks with polynomial activationscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 3; 2025: 4; 2026: 3.

18 papers.

Sort:

By area:

  1. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  2. 2023A Spectral Condition for Feature Learningcited by 101
  3. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  4. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  5. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  6. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  7. 2023Effective Theory of Transformers at Initializationcited by 21
  8. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  9. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  10. 2023Structures of Neural Network Effective Theoriescited by 13
  11. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 10
  12. 2023Feature Learning and Generalization in Deep Networks with Orthogonal Weightscited by 5
6 more
  1. 2024Principled Architecture-aware Scaling of Hyperparameterscited by 4
  2. 2024TASI Lectures on Physics for Machine Learningcited by 4
  3. 2025Function-Space Learning Ratescited by 3
  4. 2026Statistical Properties of Training & Generalizationcited by 2
  5. 2026Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Ratecited by 1
  6. 2025Deep Neural Nets as Hamiltonianscited by 0