# Convolutional Normalization

arXiv: https://arxiv.org/abs/2102.09685 · HTML: https://mlsanity.com/paper/2102.09685/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 19 Feb 2021.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis](https://mlsanity.com/paper/1701.02096/) (2017)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Iterative Normalization: Beyond Standardization towards Efficient Whitening](https://mlsanity.com/paper/1904.03441/) (2019)
- [Decorrelated Batch Normalization](https://mlsanity.com/paper/1804.08450/) (2018)
- [Depthwise Convolution is All You Need for Learning Multiple Visual Domains](https://mlsanity.com/paper/1902.00927/) (2019)

## Abstract (arXiv)

As the deep neural networks are being applied to complex tasks, the size of the networks and architecture increases and their topology becomes more complicated too. At the same time, training becomes slow and at some instances inefficient. This motivated the introduction of various normalization techniques such as Batch Normalization and Layer Normalization. The aforementioned normalization methods use arithmetic operations to compute an approximation statistics (mainly the first and second moments) of the layer's data and use it to normalize it. The aforementioned methods use plain Monte Carlo method to approximate the statistics and such method fails when approximating the statistics whose distribution is complex. Here, we propose an approach that uses weighted sum, implemented using depth-wise convolutional neural networks, to not only approximate the statistics, but to learn the coefficients of the sum.