# Iterative Normalization: Beyond Standardization towards Efficient Whitening

arXiv: https://arxiv.org/abs/1904.03441 · HTML: https://mlsanity.com/paper/1904.03441/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2019.

## Built on by (most cited first, 30 of 92)

- [RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening](https://mlsanity.com/paper/2103.15597/) (2021)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2204.02548/) (2022)
- [CLIP the Gap: A Single Domain Generalization Approach for Object Detection](https://mlsanity.com/paper/2301.05499/) (2023)
- [Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation](https://mlsanity.com/paper/2207.04892/) (2022)
- [Test-time Batch Statistics Calibration for Covariate Shift](https://mlsanity.com/paper/2110.04065/) (2021)
- [ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond](https://mlsanity.com/paper/2303.06562/) (2023)
- [Progressive Random Convolutions for Single Domain Generalization](https://mlsanity.com/paper/2304.00424/) (2023)
- [Collaborating Foundation Models for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2312.09788/) (2023)
- [Learning Content-enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentation](https://mlsanity.com/paper/2307.00371/) (2023)
- [Unbiased Faster R-CNN for Single-source Domain Generalized Object Detection](https://mlsanity.com/paper/2405.15225/) (2024)
- [SRCD: Semantic Reasoning with Compound Domains for Single-Domain Generalized Object Detection](https://mlsanity.com/paper/2307.01750/) (2023)
- [Prompt-Driven Dynamic Object-Centric Learning for Single Domain Generalization](https://mlsanity.com/paper/2402.18447/) (2024)
- [Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation](https://mlsanity.com/paper/2504.03193/) (2025)
- [Style-Hallucinated Dual Consistency Learning: A Unified Framework for Visual Domain Generalization](https://mlsanity.com/paper/2212.09068/) (2022)
- [Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2407.18568/) (2024)
- [Object-Aware Domain Generalization for Object Detection](https://mlsanity.com/paper/2312.12133/) (2023)
- [Domain Generalization via Balancing Training Difficulty and Model Capability](https://mlsanity.com/paper/2309.00844/) (2023)
- [SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training](https://mlsanity.com/paper/2412.13148/) (2024)
- [Robust Differentiable SVD](https://mlsanity.com/paper/2104.03821/) (2021)
- [Concept Learning for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.12232/) (2023)
- [VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution](https://mlsanity.com/paper/2304.01434/) (2023)
- [G-NAS: Generalizable Neural Architecture Search for Single Domain Generalization Object Detection](https://mlsanity.com/paper/2402.04672/) (2024)
- [Fast Differentiable Matrix Square Root and Inverse Square Root](https://mlsanity.com/paper/2201.12543/) (2022)
- [Few-Shot Adaptation of Pre-Trained Networks for Domain Shift](https://mlsanity.com/paper/2205.15234/) (2022)
- [Fast Differentiable Matrix Square Root](https://mlsanity.com/paper/2201.08663/) (2022)
- [Deep Interpretable Models of Theory of Mind](https://mlsanity.com/paper/2104.02938/) (2021)
- [Learning Instance-Specific Adaptation for Cross-Domain Segmentation](https://mlsanity.com/paper/2203.16530/) (2022)
- [Regularization of polynomial networks for image recognition](https://mlsanity.com/paper/2303.13896/) (2023)
- [Understanding the Failure of Batch Normalization for Transformers in NLP](https://mlsanity.com/paper/2210.05153/) (2022)

## Abstract (arXiv)

Batch Normalization (BN) is ubiquitously employed for accelerating neural network training and improving the generalization capability by performing standardization within mini-batches. Decorrelated Batch Normalization (DBN) further boosts the above effectiveness by whitening. However, DBN relies heavily on either a large batch size, or eigen-decomposition that suffers from poor efficiency on GPUs. We propose Iterative Normalization (IterNorm), which employs Newton's iterations for much more efficient whitening, while simultaneously avoiding the eigen-decomposition. Furthermore, we develop a comprehensive study to show IterNorm has better trade-off between optimization and generalization, with theoretical and experimental support. To this end, we exclusively introduce Stochastic Normalization Disturbance (SND), which measures the inherent stochastic uncertainty of samples when applied to normalization operations. With the support of SND, we provide natural explanations to several phenomena from the perspective of optimization, e.g., why group-wise whitening of DBN generally outperforms full-whitening and why the accuracy of BN degenerates with reduced batch sizes. We demonstrate the consistently improved performance of IterNorm with extensive experiments on CIFAR-10 and ImageNet over BN and DBN.