# Theoretical Insight into Batch Normalization: Data Dependant Auto-Tuning of Regularization Rate

arXiv: https://arxiv.org/abs/2209.07587 · HTML: https://mlsanity.com/paper/2209.07587/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Troubling Trends in Machine Learning Scholarship](https://mlsanity.com/paper/1807.03341/) (2018)
- [Decorrelated Batch Normalization](https://mlsanity.com/paper/1804.08450/) (2018)
- [MegDet: A Large Mini-Batch Object Detector](https://mlsanity.com/paper/1711.07240/) (2017)
- [Towards Understanding Regularization in Batch Normalization](https://mlsanity.com/paper/1809.00846/) (2018)
- [Differentiable Learning-to-Normalize via Switchable Normalization](https://mlsanity.com/paper/1806.10779/) (2018)
- [Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization](https://mlsanity.com/paper/1805.10694/) (2018)
- [Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes](https://mlsanity.com/paper/1611.04520/) (2016)

## Abstract (arXiv)

Batch normalization is widely used in deep learning to normalize intermediate activations. Deep networks suffer from notoriously increased training complexity, mandating careful initialization of weights, requiring lower learning rates, etc. These issues have been addressed by Batch Normalization (BN), by normalizing the inputs of activations to zero mean and unit standard deviation. Making this batch normalization part of the training process dramatically accelerates the training process of very deep networks. A new field of research has been going on to examine the exact theoretical explanation behind the success of BN. Most of these theoretical insights attempt to explain the benefits of BN by placing them on its influence on optimization, weight scale invariance, and regularization. Despite BN undeniable success in accelerating generalization, the gap of analytically relating the effect of BN to the regularization parameter is still missing. This paper aims to bring out the data-dependent auto-tuning of the regularization parameter by BN with analytical proofs. We have posed BN as a constrained optimization imposed on non-BN weights through which we demonstrate its data statistics dependant auto-tuning of regularization parameter. We have also given analytical proof for its behavior under a noisy input scenario, which reveals the signal vs. noise tuning of the regularization parameter. We have also substantiated our claim with empirical results from the MNIST dataset experiments.