# An Investigation into Whitening Loss for Self-supervised Learning

arXiv: https://arxiv.org/abs/2210.03586 · HTML: https://mlsanity.com/paper/2210.03586/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 68 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Graph Self-Supervised Learning: A Survey](https://mlsanity.com/paper/2103.00111/) (2021)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [Unsupervised Embedding Learning via Invariant and Spreading Instance Feature](https://mlsanity.com/paper/1904.03436/) (2019)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)

## Built on by (most cited first, 21 of 21)

- [Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism](https://mlsanity.com/paper/2303.02387/) (2023)
- [Implicit variance regularization in non-contrastive SSL](https://mlsanity.com/paper/2212.04858/) (2022)
- [The Edge of Orthogonality: A Simple View of What Makes BYOL Tick](https://mlsanity.com/paper/2302.04817/) (2023)
- [Modulate Your Spectrum in Self-Supervised Learning](https://mlsanity.com/paper/2305.16789/) (2023)
- [Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization](https://mlsanity.com/paper/2411.00392/) (2024)
- [The Common Stability Mechanism behind most Self-Supervised Learning Approaches](https://mlsanity.com/paper/2402.14957/) (2024)
- [Clustering Properties of Self-Supervised Learning](https://mlsanity.com/paper/2501.18452/) (2025)
- [Collapse-Proof Non-Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2410.04959/) (2024)
- [GEDI: GEnerative and DIscriminative Training for Self-Supervised Learning](https://mlsanity.com/paper/2212.13425/) (2022)
- [Towards the Sparseness of Projection Head in Self-Supervised Learning](https://mlsanity.com/paper/2307.08913/) (2023)
- [Training the Untrainable: Introducing Inductive Bias via Representational Alignment](https://mlsanity.com/paper/2410.20035/) (2024)
- [Dual Perspectives on Non-Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2507.01028/) (2025)
- [Semantic Concentration for Self-Supervised Dense Representations Learning](https://mlsanity.com/paper/2509.09429/) (2025)
- [Unifying Self-Supervised Clustering and Energy-Based Models](https://mlsanity.com/paper/2401.00873/) (2023)
- [T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2510.23484/) (2025)
- [Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors](https://mlsanity.com/paper/2510.04758/) (2025)
- [Training-Free Fine-Grained Semantic Segmentations in Low Data Regimes: A FungiTastic Baseline](https://mlsanity.com/paper/2605.22492/) (2026)
- [An Investigation into the Performance of Non-Contrastive Self-Supervised Learning Methods for Network Intrusion Detection](https://mlsanity.com/paper/2510.02349/) (2025)
- [Provable Subspace Identification of Nonlinear Multi-view CCA](https://mlsanity.com/paper/2602.23785/) (2026)
- [Learning a Flow to Self-Supervised Representations](https://mlsanity.com/paper/2609.29350/) (2026)
- [Which Tasks Survive Self-Supervised Learning?](https://mlsanity.com/paper/2609.38393/) (2026)

## Abstract (arXiv)

A desirable objective in self-supervised learning (SSL) is to avoid feature collapse. Whitening loss guarantees collapse avoidance by minimizing the distance between embeddings of positive pairs under the conditioning that the embeddings from different views are whitened. In this paper, we propose a framework with an informative indicator to analyze whitening loss, which provides a clue to demystify several interesting phenomena as well as a pivoting point connecting to other SSL methods. We reveal that batch whitening (BW) based methods do not impose whitening constraints on the embedding, but they only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse. Based on our analysis, we propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based methods in preventing collapse and avoids their disadvantages requiring large batch size. Experimental results on ImageNet classification and COCO object detection reveal that the proposed CW-RGP possesses a promising potential for learning good representations. The code is available at https://github.com/winci-ai/CW-RGP.