# Normalization effects on deep neural networks

arXiv: https://arxiv.org/abs/2209.01018 · HTML: https://mlsanity.com/paper/2209.01018/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 2 Sep 2022.

## Builds on (most cited first, 10 of 10)

- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Mean Field Analysis of Neural Networks: A Central Limit Theorem](https://mlsanity.com/paper/1808.09372/) (2018)
- [Scaling description of generalization with number of parameters in deep learning](https://mlsanity.com/paper/1901.01608/) (2019)
- [Understanding Deep Convolutional Networks](https://mlsanity.com/paper/1601.04920/) (2016)
- [Universal features of price formation in financial markets: perspectives from Deep Learning](https://mlsanity.com/paper/1803.06917/) (2018)
- [Deep Learning in Robotics: A Review of Recent Research](https://mlsanity.com/paper/1707.07217/) (2017)
- [Mean Field Analysis of Deep Neural Networks](https://mlsanity.com/paper/1903.04440/) (2019)
- [Asymptotics of Reinforcement Learning with Neural Networks](https://mlsanity.com/paper/1911.07304/) (2019)
- [Normalization effects on shallow neural networks and related asymptotic expansions](https://mlsanity.com/paper/2011.10487/) (2020)

## Built on by (most cited first, 6 of 6)

- [Federated Unsupervised Domain Generalization using Global and Local Alignment of Gradients](https://mlsanity.com/paper/2405.16304/) (2024)
- [Extended Deep Adaptive Input Normalization for Preprocessing Time Series Data for Neural Networks](https://mlsanity.com/paper/2310.14720/) (2023)
- [Provable Watermarking for Data Poisoning Attacks](https://mlsanity.com/paper/2510.09210/) (2025)
- [Proximal Mean Field Learning in Shallow Neural Networks](https://mlsanity.com/paper/2210.13879/) (2022)
- [AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery](https://mlsanity.com/paper/2504.01875/) (2025)
- [Deep Operator Networks for Surrogate Modeling of Cyclic Adsorption Processes with Varying Initial Conditions](https://mlsanity.com/paper/2601.09491/) (2026)

## Abstract (arXiv)

We study the effect of normalization on the layers of deep neural networks of feed-forward type. A given layer i with Ni hidden units is allowed to be normalized by 1/Ni^γi with γi∈[1/2,1] and we study the effect of the choice of the γi on the statistical behavior of the neural network's output (such as variance) as well as on the test accuracy on the MNIST data set. We find that in terms of variance of the neural network's output and test accuracy the best choice is to choose the γi's to be equal to one, which is the mean-field scaling. We also find that this is particularly true for the outer layer, in that the neural network's behavior is more sensitive in the scaling of the outer layer as opposed to the scaling of the inner layers. The mechanism for the mathematical analysis is an asymptotic expansion for the neural network's output. An important practical consequence of the analysis is that it provides a systematic and mathematically informed way to choose the learning rate hyperparameters. Such a choice guarantees that the neural network behaves in a statistically robust way as the Ni grow to infinity.