mlsanityData to 7 Oct 2026

On Feature Decorrelation in Self-Supervised Learning

arXiv 2105.00470 ↗First posted 2 May 2021cs.LG cs.AI cs.CVcited by 160

Not on the map. Why?

Abstract (arXiv)

In self-supervised representation learning, a common idea behind most of the state-of-the-art approaches is to enforce the robustness of the representations to predefined augmentations. A potential issue of this idea is the existence of completely collapsed solutions (i.e., constant features), which are typically avoided implicitly by carefully chosen implementation details. In this work, we study a relatively concise framework containing the most common components from recent approaches. We verify the existence of complete collapse and discover another reachable collapse pattern that is usually overlooked, namely dimensional collapse. We connect dimensional collapse with strong correlations between axes and consider such connection as a strong motivation for feature decorrelation (i.e., standardizing the covariance matrix). The gains from feature decorrelation are verified empirically to highlight the importance and the potential of this insight.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Builds on 43 of its 51 references that are in the map. First posted on arXiv 2 May 2021.

Builds on

43 of 51 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
31 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  4. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  5. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  6. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  7. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  8. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  9. 2020Contrastive Learning with Hard Negative Samplescited by 619
  10. 2017Large Batch Training of Convolutional Networkscited by 529
  11. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  12. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  13. 2020Whitening for Self-Supervised Representation Learningcited by 278
  14. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  15. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  16. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  17. 2020BYOL works even without batch statisticscited by 95
  18. 2015Reducing Overfitting in Deep Networks by Decorrelating Representationscited by 94
  19. 2019Iterative Normalization: Beyond Standardization towards Efficient Whiteningcited by 92
  20. 2020Contrastive Learning with Adversarial Examplescited by 91
  21. 2020AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversariescited by 90
  22. 2018Decorrelated Batch Normalizationcited by 83
  23. 2019Unsupervised Domain Adaptation using Feature-Whitening and Consensus Losscited by 67
  24. 2016Achieving Human Parity in Conversational Speech Recognitioncited by 63
  25. 2020Parametric Instance Classification for Unsupervised Visual Feature Learningcited by 41
  26. 2021Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Predictioncited by 36
  27. 2020SelfAugment: Automatic Augmentation Policies for Self-Supervised Learningcited by 32
  28. 2018Whitening and Coloring batch transform for GANscited by 27
  29. 2020Unsupervised Representation Learning by InvariancePropagationcited by 18
  30. 2020Self-supervised Pre-training with Hard Examples Improves Visual Representationscited by 12
  31. 2020Training Deep Neural Networks Without Batch Normalizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 40; 2023: 39; 2024: 39; 2025: 23; 2026: 10.

160 papers.

Sort:

By area:

  1. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  2. 2023A Cookbook of Self-Supervised Learningcited by 248
  3. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  4. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  5. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  6. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 138
  7. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  8. 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
  9. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  10. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  11. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  12. 2023Learning Semi-supervised Gaussian Mixture Models for Generalized Category Discoverycited by 72
38 more
  1. 2022Crafting Better Contrastive Views for Siamese Representation Learningcited by 67
  2. 2021Momentum Contrastive Voxel-wise Representation Learning for Semi-supervised Volumetric Medical Image Segmentationcited by 49
  3. 2021Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learningcited by 49
  4. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  5. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45
  6. 2023Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspectivecited by 43
  7. 2022Understanding Collapse in Non-Contrastive Siamese Representation Learningcited by 42
  8. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretrainingcited by 42
  9. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  10. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  11. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  12. 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
  13. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  14. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  15. 2022Self-Supervised Learning via Maximum Entropy Codingcited by 33
  16. 2024RealViformer: Investigating Attention for Real-World Video Super-Resolutioncited by 33
  17. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
  18. 2023On the Embedding Collapse when Scaling up Recommendation Modelscited by 29
  19. 2022Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Datacited by 27
  20. 2023LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architecturescited by 27
  21. 2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 25
  22. 2024Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learningcited by 25
  23. 2022Contrasting the landscape of contrastive and non-contrastive learningcited by 24
  24. 2021Towards Demystifying Representation Learning with Non-contrastive Self-supervisioncited by 23
  25. 2022Evaluating Self-Supervised Learning for Molecular Graph Embeddingscited by 22
  26. 2023A Message Passing Perspective on Learning Dynamics of Contrastive Learningcited by 22
  27. 2022An Investigation into Whitening Loss for Self-supervised Learningcited by 21
  28. 2023Whitening-based Contrastive Learning of Sentence Embeddingscited by 21
  29. 2022EquiMod: An Equivariance Module to Improve Self-Supervised Learningcited by 20
  30. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  31. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  32. 2023VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distributioncited by 17
  33. 2025Ranked Entropy Minimization for Continual Test-Time Adaptationcited by 17
  34. 2022Self-supervision through Random Segments with Autoregressive Coding (RandSAC)cited by 16
  35. 2022Consistency is the key to further mitigating catastrophic forgetting in continual learningcited by 14
  36. 2024Ads Recommendation in a Collapsed and Entangled Worldcited by 13
  37. 2022An Empirical Study on Disentanglement of Negative-free Contrastive Learningcited by 12
  38. 2023Stochastic positional embeddings improve masked image modelingcited by 12

50 most cited of 160 All 160 (JSON)