mlsanityData to 14 Oct 2026

Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods

arXiv 2205.11508 ↗First posted 23 May 2022cs.LG cs.AI cs.CVcited by 149

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations. Although SSL has recently reached a milestone: outperforming supervised methods in many modalitiesdots the theoretical foundations are limited, method-specific, and fail to provide principled design guidelines to practitioners. In this paper, we propose a unifying framework under the helm of spectral manifold learning to address those limitations. Through the course of this study, we will rigorously demonstrate that VICReg, SimCLR, BarlowTwins et al. correspond to eponymous spectral methods such as Laplacian Eigenmaps, Multidimensional Scaling et al. This unification will then allow us to obtain (i) the closed-form optimal representation for each method, (ii) the closed-form optimal network parameters in the linear regime for each method, (iii) the impact of the pairwise relations used during training on each of those quantities and on downstream task performances, and most importantly, (iv) the first theoretical bridge between contrastive and non-contrastive methods towards global and local spectral embedding methods respectively, hinting at the benefits and limitations of each. For example, (i) if the pairwise relation is aligned with the downstream task, any SSL method can be employed successfully and will recover the supervised method, but in the low data regime, VICReg's invariance hyper-parameter should be high; (ii) if the pairwise relation is misaligned with the downstream task, VICReg with small invariance hyper-parameter should be preferred over SimCLR or BarlowTwins.

Key facts. Cited by 149 papers in the map (data through 14 Oct 2026). Builds on 44 of its 117 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

44 of 117 references.

References · 44

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  9. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  12. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
32 more
  1. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  2. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  3. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  4. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  5. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  6. 2020Whitening for Self-Supervised Representation Learningcited by 279
  7. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  8. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  9. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  10. 2017Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2veccited by 198
  11. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  12. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
  13. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  14. 2018Self-Supervised Video Representation Learning with Space-Time Cubic Puzzlescited by 149
  15. 2016How to learn a graph from smooth signalscited by 146
  16. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  17. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 138
  18. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  19. 2021High Fidelity Visualization of What Your Self-Supervised Representation Knows Aboutcited by 66
  20. 2021Mean Shift for Self-Supervised Learningcited by 65
  21. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  22. 2018Spectral Inference Networks: Unifying Deep and Spectral Learningcited by 35
  23. 2020On the surprising similarities between supervised and self-supervised modelscited by 29
  24. 2018Self-supervised Learning of Geometrically Stable Features Through Probabilistic Introspectioncited by 27
  25. 2021Minimum-Distortion Embeddingcited by 26
  26. 2022Contrasting the landscape of contrastive and non-contrastive learningcited by 24
  27. 2021Towards Demystifying Representation Learning with Non-contrastive Self-supervisioncited by 23
  28. 2016WarpNet: Weakly Supervised Matching for Single-view Reconstructioncited by 22
  29. 2017A Tutorial on Canonical Correlation Methodscited by 14
  30. 2019Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearitiescited by 14
  31. 2020Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approachcited by 12
  32. 2019Generalized Clustering by Learning to Optimize Expected Normalized Cutscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 23; 2023: 46; 2024: 30; 2025: 24; 2026: 26.

149 papers.

Sort:

By area:

  1. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 273
  2. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  3. 2023A Cookbook of Self-Supervised Learningcited by 250
  4. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  5. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  6. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  7. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 58
  8. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  9. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  10. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 40
  11. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  12. 2023Simple and Asymmetric Graph Contrastive Learning without Augmentationscited by 38
38 more
  1. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  2. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  3. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 36
  4. 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
  5. 2023Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representationscited by 31
  6. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 31
  7. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 31
  8. 2026When Does LeJEPA Learn a World Model?cited by 30
  9. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  10. 2022From t-SNE to UMAP with contrastive learningcited by 28
  11. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  12. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  13. 2023Contrastive Learning Is Spectral Clustering On Similarity Graphcited by 25
  14. 2023A Graph-Theoretic Framework for Understanding Open-World Semi-Supervised Learningcited by 22
  15. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  16. 2023When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysiscited by 21
  17. 2023Improved baselines for vision-language pre-trainingcited by 20
  18. 2023Complexity Matters: Rethinking the Latent Space for Generative Modelingcited by 19
  19. 2023Towards Democratizing Joint-Embedding Self-Supervised Learningcited by 18
  20. 2024SINDER: Repairing the Singular Defects of DINOv2cited by 18
  21. 2022Neural Eigenfunctions Are Structured Representation Learnerscited by 16
  22. 2022Implicit variance regularization in non-contrastive SSLcited by 16
  23. 2023Information Flow in Self-Supervised Learningcited by 16
  24. 2025I-Con: A Unifying Framework for Representation Learningcited by 16
  25. 2023The Edge of Orthogonality: A Simple View of What Makes BYOL Tickcited by 15
  26. 2022Joint Embedding Self-Supervised Learning in the Kernel Regimecited by 12
  27. 2022Spectral Representation Learning for Conditional Moment Modelscited by 12
  28. 2022Link Prediction with Non-Contrastive Learningcited by 11
  29. 2022Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representationscited by 10
  30. 2022Localized Contrastive Learning on Graphscited by 10
  31. 2023Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusioncited by 10
  32. 2022Self-Supervised Learning Through Efference Copiescited by 9
  33. 2023Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Needcited by 9
  34. 2024Poly-View Contrastive Learningcited by 9
  35. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  36. 2026A Control Theory of Predictability in Latent World Modelscited by 9
  37. 2023MAST: Masked Augmentation Subspace Training for Generalizable Self-Supervised Priorscited by 8
  38. 2023RelationMatch: Matching In-batch Relationships for Semi-supervised Learningcited by 8

50 most cited of 149 All 149 (JSON)