mlsanityData to 14 Oct 2026

An Investigation into Whitening Loss for Self-supervised Learning

arXiv 2210.03586 ↗First posted 7 Oct 2022cs.CV cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

A desirable objective in self-supervised learning (SSL) is to avoid feature collapse. Whitening loss guarantees collapse avoidance by minimizing the distance between embeddings of positive pairs under the conditioning that the embeddings from different views are whitened. In this paper, we propose a framework with an informative indicator to analyze whitening loss, which provides a clue to demystify several interesting phenomena as well as a pivoting point connecting to other SSL methods. We reveal that batch whitening (BW) based methods do not impose whitening constraints on the embedding, but they only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse. Based on our analysis, we propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based methods in preventing collapse and avoids their disadvantages requiring large batch size. Experimental results on ImageNet classification and COCO object detection reveal that the proposed CW-RGP possesses a promising potential for learning good representations. The code is available at https://github.com/winci-ai/CW-RGP.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 41 of its 68 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

41 of 68 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  12. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
29 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  3. 2019Contrastive Multiview Codingcited by 1,301
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  5. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  6. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  7. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  8. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  9. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  10. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  11. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  12. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  13. 2021Graph Self-Supervised Learning: A Surveycited by 289
  14. 2020Whitening for Self-Supervised Representation Learningcited by 279
  15. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  16. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 232
  17. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  18. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  19. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 141
  20. 2015Optimal whitening and decorrelationcited by 97
  21. 2020BYOL works even without batch statisticscited by 95
  22. 2018Decorrelated Batch Normalizationcited by 83
  23. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  24. 2021Self-supervised Video Transformercited by 64
  25. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  26. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  27. 2018Whitening and Coloring batch transform for GANscited by 27
  28. 2022Investigating Power laws in Deep Representation Learningcited by 17
  29. 2020An Investigation into the Stochasticity of Batch Whiteningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 4; 2025: 6; 2026: 4.

21 papers.

Sort:

  1. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 33
  2. 2022Implicit variance regularization in non-contrastive SSLcited by 16
  3. 2023The Edge of Orthogonality: A Simple View of What Makes BYOL Tickcited by 15
  4. 2023Modulate Your Spectrum in Self-Supervised Learningcited by 9
  5. 2024Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularizationcited by 9
  6. 2024The Common Stability Mechanism behind most Self-Supervised Learning Approachescited by 8
  7. 2025Clustering Properties of Self-Supervised Learningcited by 8
  8. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  9. 2022GEDI: GEnerative and DIscriminative Training for Self-Supervised Learningcited by 6
  10. 2023Towards the Sparseness of Projection Head in Self-Supervised Learningcited by 4
  11. 2024Training the Untrainable: Introducing Inductive Bias via Representational Alignmentcited by 4
  12. 2025Dual Perspectives on Non-Contrastive Self-Supervised Learningcited by 4
9 more
  1. 2025Semantic Concentration for Self-Supervised Dense Representations Learningcited by 3
  2. 2023Unifying Self-Supervised Clustering and Energy-Based Modelscited by 2
  3. 2025T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learningcited by 2
  4. 2025Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priorscited by 1
  5. 2026Training-Free Fine-Grained Semantic Segmentations in Low Data Regimes: A FungiTastic Baselinecited by 1
  6. 2025An Investigation into the Performance of Non-Contrastive Self-Supervised Learning Methods for Network Intrusion Detectioncited by 0
  7. 2026Provable Subspace Identification of Nonlinear Multi-view CCAcited by 0
  8. 2026Learning a Flow to Self-Supervised Representationscited by 0
  9. 2026Which Tasks Survive Self-Supervised Learning?cited by 0