# Arithmetic-Mean μP for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets

arXiv: https://arxiv.org/abs/2510.04327 · HTML: https://mlsanity.com/paper/2510.04327/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 5 Oct 2025.

## Builds on (most cited first, 16 of 16)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [ReZero is All You Need: Fast Convergence at Large Depth](https://mlsanity.com/paper/2003.04887/) (2020)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks](https://mlsanity.com/paper/1806.05393/) (2018)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [Mean Field Residual Networks: On the Edge of Chaos](https://mlsanity.com/paper/1712.08969/) (2017)
- [How to Initialize your Network? Robust Initialization for WeightNorm & ResNets](https://mlsanity.com/paper/1906.02341/) (2019)
- [PASHA: Efficient HPO and NAS with Progressive Resource Allocation](https://mlsanity.com/paper/2207.06940/) (2022)
- [Deep Residual Networks and Weight Initialization](https://mlsanity.com/paper/1709.02956/) (2017)
- [Depth Dependence of μP Learning Rates in ReLU MLPs](https://mlsanity.com/paper/2305.07810/) (2023)
- [Is One Epoch All You Need For Multi-Fidelity Hyperparameter Optimization?](https://mlsanity.com/paper/2307.15422/) (2023)
- [The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimization](https://mlsanity.com/paper/2404.04111/) (2024)
- [Principled Architecture-aware Scaling of Hyperparameters](https://mlsanity.com/paper/2402.17440/) (2024)

## Abstract (arXiv)

Choosing an appropriate learning rate remains a key challenge in scaling depth of modern deep networks. The classical maximal update parameterization (μP) enforces a fixed per-layer update magnitude, which is well suited to homogeneous multilayer perceptrons (MLPs) but becomes ill-posed in heterogeneous architectures where residual accumulation and convolutions introduce imbalance across layers. We introduce Arithmetic-Mean μP (AM-μP), which constrains not each individual layer but the network-wide average one-step pre-activation second moment to a constant scale. Combined with a residual-aware He fan-in initialization - scaling residual-branch weights by the number of blocks (Var[W]=c/(K· mathrmfan-in)) - AM-μP yields width-robust depth laws that transfer consistently across depths. We prove that, for one- and two-dimensional convolutional networks, the maximal-update learning rate satisfies η^star(L)propto L-3/2; with zero padding, boundary effects are constant-level as Ngg k. For standard residual networks with general conv+MLP blocks, we establish η^star(L)=Theta(L-3/2), with L the minimal depth. Empirical results across a range of depths confirm the -3/2 scaling law and enable zero-shot learning-rate transfer, providing a unified and practical LR principle for convolutional and deep residual networks without additional tuning overhead.