# Group Whitening: Balancing Learning Efficiency and Representational Capacity

arXiv: https://arxiv.org/abs/2009.13333 · HTML: https://mlsanity.com/paper/2009.13333/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2020.

## Built on by (most cited first, 13 of 13)

- [Fast Differentiable Matrix Square Root and Inverse Square Root](https://mlsanity.com/paper/2201.12543/) (2022)
- [Fast Differentiable Matrix Square Root](https://mlsanity.com/paper/2201.08663/) (2022)
- [Improving GANs for Long-Tailed Data through Group Spectral Regularization](https://mlsanity.com/paper/2208.09932/) (2022)
- [Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence](https://mlsanity.com/paper/2106.03743/) (2021)
- [On the Nonlinearity of Layer Normalization](https://mlsanity.com/paper/2406.01255/) (2024)
- [Delving into the Estimation Shift of Batch Normalization in a Network](https://mlsanity.com/paper/2203.10778/) (2022)
- [Orthogonal SVD Covariance Conditioning and Latent Disentanglement](https://mlsanity.com/paper/2212.05599/) (2022)
- [Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality](https://mlsanity.com/paper/2207.02119/) (2022)
- [Batch-efficient EigenDecomposition for Small and Medium Matrices](https://mlsanity.com/paper/2207.04228/) (2022)
- [Covariance-corrected Whitening Alleviates Network Degeneration on Imbalanced Classification](https://mlsanity.com/paper/2408.17197/) (2024)
- [You Only Need End-to-End Training for Long-Tailed Recognition](https://mlsanity.com/paper/2112.05958/) (2021)
- [Domain generalization Person Re-identification on Attention-aware multi-operation strategery](https://mlsanity.com/paper/2210.10409/) (2022)
- [Parallel Layer Normalization for Universal Approximation](https://mlsanity.com/paper/2505.13142/) (2025)

## Abstract (arXiv)

Batch normalization (BN) is an important technique commonly incorporated into deep learning models to perform standardization within mini-batches. The merits of BN in improving a model's learning efficiency can be further amplified by applying whitening, while its drawbacks in estimating population statistics for inference can be avoided through group normalization (GN). This paper proposes group whitening (GW), which exploits the advantages of the whitening operation and avoids the disadvantages of normalization within mini-batches. In addition, we analyze the constraints imposed on features by normalization, and show how the batch size (group number) affects the performance of batch (group) normalized networks, from the perspective of model's representational capacity. This analysis provides theoretical guidance for applying GW in practice. Finally, we apply the proposed GW to ResNet and ResNeXt architectures and conduct experiments on the ImageNet and COCO benchmarks. Results show that GW consistently improves the performance of different architectures, with absolute gains of 1.02% ~ 1.49% in top-1 accuracy on ImageNet and 1.82% ~ 3.21% in bounding box AP on COCO.