mlsanityData to 7 Oct 2026

Understanding Dimensional Collapse in Contrastive Self-supervised Learning

arXiv 2110.09348 ↗First posted 18 Oct 2021cs.CV cs.AI cs.LGcited by 374

Not on the map. Why?

Abstract (arXiv)

Self-supervised visual representation learning aims to learn useful representations without relying on human annotations. Joint embedding approach bases on maximizing the agreement between embedding vectors from different views of the same image. Various methods have been proposed to solve the collapsing problem where all embedding vectors collapse to a trivial constant solution. Among these methods, contrastive learning prevents collapse via negative sample pairs. It has been shown that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse, whereby the embedding vectors end up spanning a lower-dimensional subspace instead of the entire available embedding space. Here, we show that dimensional collapse also happens in contrastive learning. In this paper, we shed light on the dynamics at play in contrastive learning that leads to dimensional collapse. Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR, which directly optimizes the representation space without relying on an explicit trainable projector. Experiments show that DirectCLR outperforms SimCLR with a trainable linear projector on ImageNet.

Key facts. Cited by 374 papers in the map (data through 7 Oct 2026). Builds on 31 of its 34 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

31 of 34 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
19 more
  1. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  2. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  3. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  4. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  5. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  8. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  9. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  10. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  11. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  12. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  13. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  14. 2020Implicit Gradient Regularizationcited by 145
  15. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  16. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  17. 2021Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samplescited by 114
  18. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  19. 2020Implicit Rank-Minimizing Autoencodercited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 65; 2023: 80; 2024: 79; 2025: 68; 2026: 79.

374 papers.

Sort:

By area:

  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  2. 2023A Cookbook of Self-Supervised Learningcited by 248
  3. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  4. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  5. 2022A Self-Supervised Descriptor for Image Copy Detectioncited by 175
  6. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  7. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  8. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  9. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  10. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  11. 2023What Do Self-Supervised Vision Transformers Learn?cited by 90
  12. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
38 more
  1. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  2. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  3. 2023Learning Semi-supervised Gaussian Mixture Models for Generalized Category Discoverycited by 72
  4. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 63
  5. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  6. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 57
  7. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  8. 2022Spectral Feature Augmentation for Graph Contrastive Learning and Beyondcited by 49
  9. 2024EEG2Rep: Enhancing Self-supervised EEG Representation Through Informative Masked Inputscited by 48
  10. 2022Robust Contrastive Learning against Noisy Viewscited by 45
  11. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45
  12. 2023Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspectivecited by 43
  13. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  14. 2023M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimizationcited by 43
  15. 2022Understanding Collapse in Non-Contrastive Siamese Representation Learningcited by 42
  16. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretrainingcited by 42
  17. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 41
  18. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  19. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  20. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  21. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 39
  22. 2022Understanding and Improving the Role of Projection Head in Self-Supervised Learningcited by 36
  23. 2023Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-trainingcited by 36
  24. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  25. 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
  26. 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 35
  27. 2022ProtoVAE: A Trustworthy Self-Explainable Prototypical Variational Modelcited by 34
  28. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  29. 2023Reverse Engineering Self-Supervised Learningcited by 33
  30. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 33
  31. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
  32. 2023Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representationscited by 31
  33. 2024Artificial General Intelligence (AGI)-Native Wireless Systems: A Journey Beyond 6Gcited by 30
  34. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 30
  35. 2023On the Embedding Collapse when Scaling up Recommendation Modelscited by 29
  36. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  37. 2023LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architecturescited by 27
  38. 2024Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Datacited by 27

50 most cited of 374 All 374 (JSON)