mlsanityData to 14 Oct 2026

On the duality between contrastive and non-contrastive self-supervised learning

arXiv 2206.02574 ↗First posted 3 Jun 2022cs.LG cs.AI cs.CVcited by 108

Not on the map. Why?

Abstract (arXiv)

Recent approaches in self-supervised learning of image representations can be categorized into different families of methods and, in particular, can be divided into contrastive and non-contrastive approaches. While differences between the two families have been thoroughly discussed to motivate new approaches, we focus more on the theoretical similarities between them. By designing contrastive and covariance based non-contrastive criteria that can be related algebraically and shown to be equivalent under limited assumptions, we show how close those families can be. We further study popular methods and introduce variations of them, allowing us to relate this theoretical result to current practices and show the influence (or lack thereof) of design choices on downstream performance. Motivated by our equivalence result, we investigate the low performance of SimCLR and show how it can match VICReg's with careful hyperparameter tuning, improving significantly over known baselines. We also challenge the popular assumption that non-contrastive methods need large output dimensions. Our theoretical and quantitative results suggest that the numerical gaps between contrastive and non-contrastive methods in certain regimes can be closed given better network design choices and hyperparameter tuning. The evidence shows that unifying different SOTA methods is an important direction to build a better understanding of self-supervised learning.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

38 of 52 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
26 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  2. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  3. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  4. 2017Large Batch Training of Convolutional Networkscited by 535
  5. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  6. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  7. 2020Whitening for Self-Supervised Representation Learningcited by 279
  8. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  9. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  10. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  11. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  12. 2021Decoupled Contrastive Learningcited by 143
  13. 2020Intriguing Properties of Contrastive Lossescited by 130
  14. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  15. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  16. 2021Watermarking Images in Self-Supervised Latent Spacescited by 94
  17. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  18. 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 67
  19. 2022Mugs: A Multi-Granular Self-Supervised Learning Frameworkcited by 54
  20. 2022Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCocited by 52
  21. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  22. 2022Neural Manifold Clustering and Embeddingcited by 44
  23. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  24. 2021Compressive Visual Representationscited by 41
  25. 2021A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learningcited by 26
  26. 2022Contrasting the landscape of contrastive and non-contrastive learningcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 13; 2023: 37; 2024: 23; 2025: 20; 2026: 15.

108 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 250
  2. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  3. 2023AnyLoc: Towards Universal Visual Place Recognitioncited by 163
  4. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 160
  5. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  6. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  7. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 72
  8. 2025Scaling Language-Free Visual Representation Learningcited by 66
  9. 2023LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matchingcited by 60
  10. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 58
  11. 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 57
  12. 2023Self-supervised learning of Split Invariant Equivariant representationscited by 44
38 more
  1. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  2. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 40
  3. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  4. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  5. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  6. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 36
  7. 2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29
  8. 2023Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentationcited by 27
  9. 2024Do Generated Data Always Help Contrastive Learning?cited by 25
  10. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  11. 2022VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignmentcited by 20
  12. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20
  13. 2024UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learningcited by 18
  14. 2023Self-Supervised Disentanglement by Leveraging Structure in Data Augmentationscited by 17
  15. 2022Neural Eigenfunctions Are Structured Representation Learnerscited by 16
  16. 2022Implicit variance regularization in non-contrastive SSLcited by 16
  17. 2023MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillationcited by 16
  18. 2023Information Flow in Self-Supervised Learningcited by 16
  19. 2023Domain-specific optimization and diverse evaluation of self-supervised models for histopathologycited by 11
  20. 2023Combating Representation Learning Disparity with Geometric Harmonizationcited by 11
  21. 2024LDReg: Local Dimensionality Regularized Self-Supervised Learningcited by 11
  22. 2026Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Modelscited by 11
  23. 2022Bag of Image Patch Embedding Behind the Success of Self-Supervised Learningcited by 10
  24. 2022Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representationscited by 10
  25. 2026Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representationscited by 10
  26. 2023Modulate Your Spectrum in Self-Supervised Learningcited by 9
  27. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  28. 2024Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representationscited by 8
  29. 2024The Hidden Pitfalls of the Cosine Similarity Losscited by 7
  30. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  31. 2024HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithmscited by 7
  32. 2024Understanding the Role of Equivariance in Self-supervised Learningcited by 7
  33. 2022Minimalistic Unsupervised Learning with the Sparse Manifold Transformcited by 6
  34. 2022GEDI: GEnerative and DIscriminative Training for Self-Supervised Learningcited by 6
  35. 2022DyG2Vec: Efficient Representation Learning for Dynamic Graphscited by 5
  36. 2023Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shiftcited by 5
  37. 2023Harnessing small projectors and multiple views for efficient vision pretrainingcited by 5
  38. 2024A Probabilistic Model Behind Self-Supervised Learningcited by 5

50 most cited of 108 All 108 (JSON)