mlsanityData to 14 Oct 2026

Iterative Normalization: Beyond Standardization towards Efficient Whitening

arXiv 1904.03441 ↗First posted 6 Apr 2019cs.CV cs.LGcited by 92

Not on the map. Why?

Abstract (arXiv)

Batch Normalization (BN) is ubiquitously employed for accelerating neural network training and improving the generalization capability by performing standardization within mini-batches. Decorrelated Batch Normalization (DBN) further boosts the above effectiveness by whitening. However, DBN relies heavily on either a large batch size, or eigen-decomposition that suffers from poor efficiency on GPUs. We propose Iterative Normalization (IterNorm), which employs Newton's iterations for much more efficient whitening, while simultaneously avoiding the eigen-decomposition. Furthermore, we develop a comprehensive study to show IterNorm has better trade-off between optimization and generalization, with theoretical and experimental support. To this end, we exclusively introduce Stochastic Normalization Disturbance (SND), which measures the inherent stochastic uncertainty of samples when applied to normalization operations. With the support of SND, we provide natural explanations to several phenomena from the perspective of optimization, e.g., why group-wise whitening of DBN generally outperforms full-whitening and why the accuracy of BN degenerates with reduced batch sizes. We demonstrate the consistently improved performance of IterNorm with extensive experiments on CIFAR-10 and ImageNet over BN and DBN.

Key facts. Cited by 92 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 13; 2023: 20; 2024: 16; 2025: 25; 2026: 5.

92 papers.

Sort:

By area:

  1. 2021RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whiteningcited by 215
  2. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  3. 2022Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentationcited by 89
  4. 2023CLIP the Gap: A Single Domain Generalization Approach for Object Detectioncited by 87
  5. 2022Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentationcited by 70
  6. 2021Test-time Batch Statistics Calibration for Covariate Shiftcited by 50
  7. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  8. 2023Progressive Random Convolutions for Single Domain Generalizationcited by 34
  9. 2023Collaborating Foundation Models for Domain Generalized Semantic Segmentationcited by 34
  10. 2023Learning Content-enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentationcited by 30
  11. 2024Unbiased Faster R-CNN for Single-source Domain Generalized Object Detectioncited by 29
  12. 2023SRCD: Semantic Reasoning with Compound Domains for Single-Domain Generalized Object Detectioncited by 28
38 more
  1. 2024Prompt-Driven Dynamic Object-Centric Learning for Single Domain Generalizationcited by 24
  2. 2025Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentationcited by 24
  3. 2022Style-Hallucinated Dual Consistency Learning: A Unified Framework for Visual Domain Generalizationcited by 22
  4. 2024Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentationcited by 22
  5. 2023Object-Aware Domain Generalization for Object Detectioncited by 21
  6. 2023Domain Generalization via Balancing Training Difficulty and Model Capabilitycited by 20
  7. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  8. 2021Robust Differentiable SVDcited by 19
  9. 2023Concept Learning for Interpretable Multi-Agent Reinforcement Learningcited by 19
  10. 2023VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distributioncited by 17
  11. 2024G-NAS: Generalizable Neural Architecture Search for Single Domain Generalization Object Detectioncited by 17
  12. 2022Fast Differentiable Matrix Square Root and Inverse Square Rootcited by 16
  13. 2022Few-Shot Adaptation of Pre-Trained Networks for Domain Shiftcited by 16
  14. 2022Fast Differentiable Matrix Square Rootcited by 14
  15. 2021Deep Interpretable Models of Theory of Mindcited by 13
  16. 2022Learning Instance-Specific Adaptation for Cross-Domain Segmentationcited by 13
  17. 2023Regularization of polynomial networks for image recognitioncited by 13
  18. 2022Understanding the Failure of Batch Normalization for Transformers in NLPcited by 12
  19. 2024Concept-Attention Whitening for Interpretable Skin Lesion Diagnosiscited by 11
  20. 2024PhysAug: A Physical-guided and Frequency-based Data Augmentation for Single-Domain Generalized Object Detectioncited by 11
  21. 2021Stochastic Whitening Batch Normalizationcited by 9
  22. 2023Real-time Controllable Denoising for Image and Videocited by 9
  23. 2023Modulate Your Spectrum in Self-Supervised Learningcited by 9
  24. 2025Gradient Multi-Normalization for Stateless and Scalable LLM Trainingcited by 9
  25. 2022Semantic Self-adaptation: Enhancing Generalization with a Single Samplecited by 8
  26. 2022Normalization Perturbation: A Simple Domain Generalization Method for Real-World Domain Shiftscited by 8
  27. 2024Exploring Semantic Consistency and Style Diversity for Domain Generalized Semantic Segmentationcited by 8
  28. 2023Pulling Target to Source: A New Perspective on Domain Adaptive Semantic Segmentationcited by 7
  29. 2024Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentationcited by 7
  30. 2024Correlations Are Ruining Your Gradient Descentcited by 7
  31. 2025Stronger, Steadier & Superior: Geometric Consistency in Depth VFM Forges Domain Generalized Semantic Segmentationcited by 7
  32. 2025Generalized Diffusion Detector: Mining Robust Features from Diffusion Models for Domain-Generalized Detectioncited by 6
  33. 2025Style Evolving along Chain-of-Thought for Unknown-Domain Object Detectioncited by 6
  34. 2025On the Importance of Gaussianizing Representationscited by 6
  35. 2022Orthogonal SVD Covariance Conditioning and Latent Disentanglementcited by 5
  36. 2023Benchmarking and Enhancing Disentanglement in Concept-Residual Modelscited by 5
  37. 2024Phrase Grounding-based Style Transfer for Single-Domain Generalized Object Detectioncited by 5
  38. 2024Whitening Consistently Improves Self-Supervised Learningcited by 5

50 most cited of 92 All 92 (JSON)