# Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

arXiv: https://arxiv.org/abs/1603.01431 · HTML: https://mlsanity.com/paper/1603.01431/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2016.

## Built on by (most cited first, 20 of 20)

- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [Characterizing signal propagation to close the performance gap in unnormalized ResNets](https://mlsanity.com/paper/2101.08692/) (2021)
- [Normalization and effective learning rates in reinforcement learning](https://mlsanity.com/paper/2407.01800/) (2024)
- [Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning](https://mlsanity.com/paper/2106.05956/) (2021)
- ["BNN - BN = ?": Training Binary Neural Networks without Batch Normalization](https://mlsanity.com/paper/2104.08215/) (2021)
- [Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning](https://mlsanity.com/paper/2101.07525/) (2021)
- [Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence](https://mlsanity.com/paper/2106.03743/) (2021)
- [Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training](https://mlsanity.com/paper/2106.03640/) (2021)
- [Information Geometrically Generalized Covariate Shift Adaptation](https://mlsanity.com/paper/2304.09387/) (2023)
- [On the Importance of Gaussianizing Representations](https://mlsanity.com/paper/2505.00685/) (2025)
- [Rethinking Layer-wise Model Merging through Chain of Merges](https://mlsanity.com/paper/2508.21421/) (2025)
- [Revisiting Batch Norm Initialization](https://mlsanity.com/paper/2110.13989/) (2021)
- [Batch Layer Normalization, A new normalization layer for CNNs and RNN](https://mlsanity.com/paper/2209.08898/) (2022)
- [A Survey of Historical Learning: Learning Models with Learning History](https://mlsanity.com/paper/2303.12992/) (2023)
- [Noise Injection as a Probe of Deep Learning Dynamics](https://mlsanity.com/paper/2210.13599/) (2022)
- [BWCP: Probabilistic Learning-to-Prune Channels for ConvNets via Batch Whitening](https://mlsanity.com/paper/2105.06423/) (2021)
- [A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization](https://mlsanity.com/paper/2112.12299/) (2021)
- [BCN: Batch Channel Normalization for Image Classification](https://mlsanity.com/paper/2312.00596/) (2023)
- [Rethinking Activation Functions: Reducing Bias Can Improve Learning](https://mlsanity.com/paper/2610.14500/) (2026)

## Abstract (arXiv)

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of mean and standard deviation of input (distribution) to hidden layers inaccurate for validation due to shifting parameter values (especially during initial training epochs). Also, BN cannot be used with batch-size 1 during training. We address these drawbacks by proposing a non-adaptive normalization technique for removing internal covariate shift, that we call Normalization Propagation. Our approach does not depend on batch statistics, but rather uses a data-independent parametric estimate of mean and standard-deviation in every layer thus being computationally faster compared with BN. We exploit the observation that the pre-activation before Rectified Linear Units follow Gaussian distribution in deep networks, and that once the first and second order statistics of any given dataset are normalized, we can forward propagate this normalization without the need for recalculating the approximate statistics for hidden layers.