# Improving Generalization of Batch Whitening by Convolutional Unit Optimization

arXiv: https://arxiv.org/abs/2108.10629 · HTML: https://mlsanity.com/paper/2108.10629/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 24 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Optimal whitening and decorrelation](https://mlsanity.com/paper/1512.00809/) (2015)
- [Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions](https://mlsanity.com/paper/1711.08141/) (2017)
- [Iterative Normalization: Beyond Standardization towards Efficient Whitening](https://mlsanity.com/paper/1904.03441/) (2019)
- [Decorrelated Batch Normalization](https://mlsanity.com/paper/1804.08450/) (2018)
- [Bayesian Uncertainty Estimation for Batch Normalized Deep Networks](https://mlsanity.com/paper/1802.06455/) (2018)
- [Natural Neural Networks](https://mlsanity.com/paper/1507.00210/) (2015)
- [All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1903.05285/) (2019)
- [Whitening and Coloring batch transform for GANs](https://mlsanity.com/paper/1806.00420/) (2018)
- [Constructing Fast Network through Deconstruction of Convolution](https://mlsanity.com/paper/1806.07370/) (2018)
- [An Investigation into the Stochasticity of Batch Whitening](https://mlsanity.com/paper/2003.12327/) (2020)
- [Network Deconvolution](https://mlsanity.com/paper/1905.11926/) (2019)

## Built on by (most cited first, 3 of 3)

- [Fast Differentiable Matrix Square Root and Inverse Square Root](https://mlsanity.com/paper/2201.12543/) (2022)
- [Controllable Feature Whitening for Hyperparameter-Free Bias Mitigation](https://mlsanity.com/paper/2507.20284/) (2025)
- [Localization using Multi-Focal Spatial Attention for Masked Face Recognition](https://mlsanity.com/paper/2305.01905/) (2023)

## Abstract (arXiv)

Batch Whitening is a technique that accelerates and stabilizes training by transforming input features to have a zero mean (Centering) and a unit variance (Scaling), and by removing linear correlation between channels (Decorrelation). In commonly used structures, which are empirically optimized with Batch Normalization, the normalization layer appears between convolution and activation function. Following Batch Whitening studies have employed the same structure without further analysis; even Batch Whitening was analyzed on the premise that the input of a linear layer is whitened. To bridge the gap, we propose a new Convolutional Unit that is in line with the theory, and our method generally improves the performance of Batch Whitening. Moreover, we show the inefficacy of the original Convolutional Unit by investigating rank and correlation of features. As our method is employable off-the-shelf whitening modules, we use Iterative Normalization (IterNorm), the state-of-the-art whitening module, and obtain significantly improved performance on five image classification datasets: CIFAR-10, CIFAR-100, CUB-200-2011, Stanford Dogs, and ImageNet. Notably, we verify that our method improves stability and performance of whitening when using large learning rate, group size, and iteration number.