# Depth Dependence of μP Learning Rates in ReLU MLPs

arXiv: https://arxiv.org/abs/2305.07810 · HTML: https://mlsanity.com/paper/2305.07810/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 9 references that are in the map.
- First posted on arXiv 13 May 2023.

## Builds on (most cited first, 7 of 7)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [Loss landscapes and optimization in over-parameterized non-linear systems and neural networks](https://mlsanity.com/paper/2003.00307/) (2020)
- [Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?](https://mlsanity.com/paper/1801.03744/) (2018)
- [A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks](https://mlsanity.com/paper/2001.11443/) (2020)

## Built on by (most cited first, 9 of 9)

- [Scalable Optimization in the Modular Norm](https://mlsanity.com/paper/2405.14813/) (2024)
- [Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit](https://mlsanity.com/paper/2309.16620/) (2023)
- [Super Consistency of Neural Network Landscapes and Learning Rate Transfer](https://mlsanity.com/paper/2402.17457/) (2024)
- [The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks](https://mlsanity.com/paper/2311.18718/) (2023)
- [Principled Architecture-aware Scaling of Hyperparameters](https://mlsanity.com/paper/2402.17440/) (2024)
- [Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks](https://mlsanity.com/paper/2602.07494/) (2026)
- [Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum](https://mlsanity.com/paper/2609.39595/) (2026)
- [Arithmetic-Mean μP for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets](https://mlsanity.com/paper/2510.04327/) (2025)
- [Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks](https://mlsanity.com/paper/2607.07884/) (2026)

## Abstract (arXiv)

In this short note we consider random fully connected ReLU networks of width n and depth L equipped with a mean-field weight initialization. Our purpose is to study the dependence on n and L of the maximal update (μP) learning rate, the largest learning rate for which the mean squared change in pre-activations after one step of gradient descent remains uniformly bounded at large n,L. As in prior work on μP of Yang et. al., we find that this maximal update learning rate is independent of n for all but the first and last layer weights. However, we find that it has a non-trivial dependence of L, scaling like L-3/2.