# Training Faster by Separating Modes of Variation in Batch-normalized Models

arXiv: https://arxiv.org/abs/1806.02892 · HTML: https://mlsanity.com/paper/1806.02892/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2018.

## Built on by (most cited first, 7 of 7)

- [Un-Mixing Test-Time Normalization Statistics: Combatting Label Temporal Correlation](https://mlsanity.com/paper/2401.08328/) (2024)
- [PSO-Convolutional Neural Networks with Heterogeneous Learning Rate](https://mlsanity.com/paper/2205.10456/) (2022)
- [Context Normalization Layer with Applications](https://mlsanity.com/paper/2303.07651/) (2023)
- [Enhancing Neural Network Representations with Prior Knowledge-Based Normalization](https://mlsanity.com/paper/2403.16798/) (2024)
- [Supervised Batch Normalization](https://mlsanity.com/paper/2405.17027/) (2024)
- [Unsupervised Adaptive Normalization](https://mlsanity.com/paper/2409.04757/) (2024)
- [Adaptative Context Normalization: A Boost for Deep Learning in Image Processing](https://mlsanity.com/paper/2409.04759/) (2024)

## Abstract (arXiv)

Batch Normalization (BN) is essential to effectively train state-of-the-art deep Convolutional Neural Networks (CNN). It normalizes inputs to the layers during training using the statistics of each mini-batch. In this work, we study BN from the viewpoint of Fisher kernels. We show that assuming samples within a mini-batch are from the same probability density function, then BN is identical to the Fisher vector of a Gaussian distribution. That means BN can be explained in terms of kernels that naturally emerge from the probability density function of the underlying data distribution. However, given the rectifying non-linearities employed in CNN architectures, distribution of inputs to the layers show heavy tail and asymmetric characteristics. Therefore, we propose approximating underlying data distribution not with one, but a mixture of Gaussian densities. Deriving Fisher vector for a Gaussian Mixture Model (GMM), reveals that BN can be improved by independently normalizing with respect to the statistics of disentangled sub-populations. We refer to our proposed soft piecewise version of BN as Mixture Normalization (MN). Through extensive set of experiments on CIFAR-10 and CIFAR-100, we show that MN not only effectively accelerates training image classification and Generative Adversarial networks, but also reaches higher quality models.