# Beyond Folklore: A Scaling Calculus for the Design and Initialization of ReLU Networks

arXiv: https://arxiv.org/abs/1906.04267 · HTML: https://mlsanity.com/paper/1906.04267/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2019.

## Built on by (most cited first, 4 of 4)

- [Why Gradients Rapidly Increase Near the End of Training](https://mlsanity.com/paper/2506.02285/) (2025)
- [Principled Architecture-aware Scaling of Hyperparameters](https://mlsanity.com/paper/2402.17440/) (2024)
- [Stabilizing Spiking Neuron Training](https://mlsanity.com/paper/2202.00282/) (2022)
- [Stabilizing RNN Gradients through Pre-training](https://mlsanity.com/paper/2308.12075/) (2023)

## Abstract (arXiv)

We propose a system for calculating a "scaling constant" for layers and weights of neural networks. We relate this scaling constant to two important quantities that relate to the optimizability of neural networks, and argue that a network that is "preconditioned" via scaling, in the sense that all weights have the same scaling constant, will be easier to train. This scaling calculus results in a number of consequences, among them the fact that the geometric mean of the fan-in and fan-out, rather than the fan-in, fan-out, or arithmetic mean, should be used for the initialization of the variance of weights in a neural network. Our system allows for the off-line design & engineering of ReLU neural networks, potentially replacing blind experimentation.