Data to 7 Oct 2026

Whitening for Self-Supervised Representation Learning

arXiv 2007.06346 ↗First posted 13 Jul 2020cs.LG cs.CV stat.MLcited by 278

Not on the map. Why?

Abstract (arXiv)

Most of the current self-supervised representation learning (SSL) methods are based on the contrastive loss and the instance-discrimination task, where augmented versions of the same image instance ("positives") are contrasted with instances extracted from other images ("negatives"). For the learning to be effective, many negatives should be compared with a positive pair, which is computationally demanding. In this paper, we propose a different direction and a new loss function for SSL, which is based on the whitening of the latent-space features. The whitening operation has a "scattering" effect on the batch samples, avoiding degenerate solutions where all the sample representations collapse to a single point. Our solution does not require asymmetric networks and it is conceptually simple. Moreover, since negatives are not needed, we can extract multiple positive pairs from the same image instance. The source code of the method and of all the experiments is available at: https://github.com/htdt/self-supervised.

Key facts. Cited by 278 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 75; 2023: 65; 2024: 48; 2025: 26; 2026: 41.

278 papers.

Sort:

By area:

2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
2023A Cookbook of Self-Supervised Learningcited by 248
2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
2021On Feature Decorrelation in Self-Supervised Learningcited by 160
2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
2021Self-Supervised Models are Continual Learnerscited by 149
2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145

38 more

2021Decoupled Contrastive Learningcited by 141
2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 138
2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 128
2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
2021Efficient Training of Visual Transformers with Small Datasetscited by 107
2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
2021Backdoor Attacks on Self-Supervised Learningcited by 89
2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
2022Adversarial Masking for Self-Supervised Learningcited by 78
2024Learning and Leveraging World Models in Visual Representation Learningcited by 71
2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
2022Bootstrapped Masked Autoencoders for Vision BERT Pretrainingcited by 60
2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 53
2022Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCocited by 52
2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
2022DiRA: Discriminative, Restorative, and Adversarial Learning for Self-supervised Medical Image Analysiscited by 48
2023Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Datacited by 44
2022Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structurecited by 43
2023Self-supervised learning of Split Invariant Equivariant representationscited by 41
2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
2023Non-Contrastive Unsupervised Learning of Physiological Signals from Videocited by 37
2021Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Predictioncited by 36
2022Understanding and Improving the Role of Projection Head in Self-Supervised Learningcited by 36
2022Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalizationcited by 35
2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
2022Self-Supervised Learning via Maximum Entropy Codingcited by 33
2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
2023Downstream-agnostic Adversarial Examplescited by 32
2022Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenescited by 31
2022Architecture-Agnostic Masked Image Modeling -- From ViT back to CNNcited by 31
2023Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representationscited by 31
2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29

50 most cited of 278 All 278 (JSON)