mlsanityData to 7 Oct 2026

Decorrelated Batch Normalization

arXiv 1804.08450 ↗First posted 23 Apr 2018cs.CV cs.LG stat.MLcited by 83

Not on the map. Why?

Abstract (arXiv)

Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find that PCA whitening causes a problem we call stochastic axis swapping, which is detrimental to learning. We show that ZCA whitening does not suffer from this problem, permitting successful learning. DBN retains the desirable qualities of BN and further improves BN's optimization efficiency and generalization ability. We design comprehensive experiments to show that DBN can improve the performance of BN on multilayer perceptrons and convolutional neural networks. Furthermore, we consistently improve the accuracy of residual networks on CIFAR-10, CIFAR-100, and ImageNet.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 22; 2023: 16; 2024: 10; 2025: 10; 2026: 6.

83 papers.

Sort:

By area:

  1. 2021RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whiteningcited by 213
  2. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  3. 2022WildNet: Learning Domain Generalized Semantic Segmentation from the Wildcited by 75
  4. 2021Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentationcited by 56
  5. 2021Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learningcited by 49
  6. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45
  7. 2023In Search of Lost Online Test-time Adaptation: A Surveycited by 43
  8. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  9. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  10. 2023Why Batch Normalization Damage Federated Learning on Non-IID Data?cited by 29
  11. 2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27
  12. 2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
38 more
  1. 2021Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling?cited by 24
  2. 2021Augmenting Deep Classifiers with Polynomial Neural Networkscited by 21
  3. 2022An Investigation into Whitening Loss for Self-supervised Learningcited by 21
  4. 2023Whitening-based Contrastive Learning of Sentence Embeddingscited by 21
  5. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  6. 2021Robust Differentiable SVDcited by 19
  7. 2023VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distributioncited by 17
  8. 2022Fast Differentiable Matrix Square Root and Inverse Square Rootcited by 16
  9. 2022Fast Differentiable Matrix Square Rootcited by 14
  10. 2023Stable Anisotropic Regularizationcited by 12
  11. 2022Domain Generalization with Correlated Style Uncertaintycited by 10
  12. 2021Stochastic Whitening Batch Normalizationcited by 9
  13. 2022Constrained Parameter Inference as a Principle for Learningcited by 9
  14. 2022Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representationscited by 9
  15. 2023Real-time Controllable Denoising for Image and Videocited by 9
  16. 2023Modulate Your Spectrum in Self-Supervised Learningcited by 9
  17. 2023On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learningcited by 8
  18. 2022Delving into the Estimation Shift of Batch Normalization in a Networkcited by 7
  19. 2024Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentationcited by 7
  20. 2024Efficient Deep Learning with Decorrelated Backpropagationcited by 7
  21. 2024Correlations Are Ruining Your Gradient Descentcited by 7
  22. 2024Gradient-Free Training of Recurrent Neural Networks using Random Perturbationscited by 6
  23. 2025On the Importance of Gaussianizing Representationscited by 6
  24. 2022Robustness to corruption in pre-trained Bayesian neural networkscited by 5
  25. 2022Orthogonal SVD Covariance Conditioning and Latent Disentanglementcited by 5
  26. 2023Hierarchical Semantic Tree Concept Whitening for Interpretable Image Classificationcited by 5
  27. 2023Revisiting LARS for Large Batch Training Generalization of Neural Networkscited by 5
  28. 2022Improving Covariance Conditioning of the SVD Meta-layer by Orthogonalitycited by 4
  29. 2022Interneurons accelerate learning dynamics in recurrent neural networks for statistical adaptationcited by 4
  30. 2026PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Trainingcited by 4
  31. 2021Improving Generalization of Batch Whitening by Convolutional Unit Optimizationcited by 3
  32. 2021Computer Vision Self-supervised Learning Methods on Time Seriescited by 3
  33. 2021Differentiable Gaussianization Layers for Inverse Problems Regularized by Deep Generative Modelscited by 3
  34. 2025Adapt in the Wild: Test-Time Entropy Minimization with Sharpness and Feature Regularizationcited by 3
  35. 2021Exploiting Invariance in Training Deep Neural Networkscited by 2
  36. 2021Learnable Adaptive Cosine Estimator (LACE) for Image Classificationcited by 2
  37. 2022Batch-efficient EigenDecomposition for Small and Medium Matricescited by 2
  38. 2023Self-Supervised Node Representation Learning via Node-to-Neighbourhood Alignmentcited by 2

50 most cited of 83 All 83 (JSON)