mlsanityData to 7 Oct 2026

Depth Dependence of μP Learning Rates in ReLU MLPs

arXiv 2305.07810 ↗First posted 13 May 2023cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

In this short note we consider random fully connected ReLU networks of width n and depth L equipped with a mean-field weight initialization. Our purpose is to study the dependence on n and L of the maximal update (μP) learning rate, the largest learning rate for which the mean squared change in pre-activations after one step of gradient descent remains uniformly bounded at large n,L. As in prior work on μP of Yang et. al., we find that this maximal update learning rate is independent of n for all but the first and last layer weights. However, we find that it has a non-trivial dependence of L, scaling like L-3/2.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 7 of its 9 references that are in the map. First posted on arXiv 13 May 2023.

Builds on

7 of 9 references.

References · 7

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  5. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  6. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  7. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 1; 2026: 3.

9 papers.

  1. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  2. 2024Scalable Optimization in the Modular Normcited by 64
  3. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  4. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  5. 2024Principled Architecture-aware Scaling of Hyperparameterscited by 4
  6. 2026Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networkscited by 1
  7. 2025Arithmetic-Mean μP for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNetscited by 0
  8. 2026Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networkscited by 0
  9. 2026Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentumcited by 0