mlsanityData to 7 Oct 2026

Contrasting the landscape of contrastive and non-contrastive learning

arXiv 2203.15702 ↗First posted 29 Mar 2022cs.LG cs.CV stat.MLcited by 24

Not on the map. Why?

Abstract (arXiv)

A lot of recent advances in unsupervised feature learning are based on designing features which are invariant under semantic data augmentations. A common way to do this is contrastive learning, which uses positive and negative samples. Some recent works however have shown promising results for non-contrastive learning, which does not require negative samples. However, the non-contrastive losses have obvious "collapsed" minima, in which the encoders output a constant feature embedding, independent of the input. A folk conjecture is that so long as these collapsed solutions are avoided, the produced feature representations should be good. In our paper, we cast doubt on this story: we show through theoretical results and controlled experiments that even on simple data models, non-contrastive losses have a preponderance of non-collapsed bad minima. Moreover, we show that the training process does not avoid these minima.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 29 Mar 2022.

Builds on

30 of 36 references.

References · 30

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  12. 2019Contrastive Multiview Codingcited by 1,297
18 more
  1. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  2. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  3. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  4. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  5. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  6. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  7. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  8. 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
  9. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  10. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 231
  11. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  12. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  13. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  14. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  15. 2020BYOL works even without batch statisticscited by 95
  16. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  17. 2015Simple, Efficient, and Neural Algorithms for Sparse Codingcited by 37
  18. 2021Towards Demystifying Representation Learning with Non-contrastive Self-supervisioncited by 23

Built on by

Cited by year

Citing papers per year: 2022: 9; 2023: 11; 2024: 3; 2025: 1.

24 papers.

Sort:

  1. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  2. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  3. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  4. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  5. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  6. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  7. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  8. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  9. 2023Vision-Language Dataset Distillationcited by 18
  10. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  11. 2023Unleashing the Power of Self-Supervised Image Denoising: A Comprehensive Reviewcited by 17
  12. 2023Information Flow in Self-Supervised Learningcited by 16
12 more
  1. 2022Implicit variance regularization in non-contrastive SSLcited by 15
  2. 2023Unsupervised Representation Learning for Time Series: A Reviewcited by 14
  3. 2024Localizing Memorization in SSL Vision Encoderscited by 9
  4. 2022Analyzing Data-Centric Properties for Graph Contrastive Learningcited by 7
  5. 2022Empirical Evaluation and Theoretical Analysis for Representation Learning: A Surveycited by 5
  6. 2023On-ramp and Off-ramp Traffic Flows Estimation Based on A Data-driven Transfer Learning Frameworkcited by 3
  7. 2023A Transfer Learning Framework for Proactive Ramp Metering Performance Assessmentcited by 3
  8. 2023Representation Learning Dynamics of Self-Supervised Modelscited by 3
  9. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
  10. 2022TriNet: stabilizing self-supervised learning from complete or slow collapse on ASRcited by 1
  11. 2024When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?cited by 1
  12. 2025A Probabilistic Model for Non-Contrastive Learningcited by 0