mlsanityData to 7 Oct 2026

The Power of Contrast for Feature Learning: A Theoretical Analysis

arXiv 2110.02473 ↗First posted 6 Oct 2021cs.LG stat.MLcited by 53

Not on the map. Why?

Abstract (arXiv)

Contrastive learning has achieved state-of-the-art performance in various self-supervised learning tasks and even outperforms its supervised counterpart. Despite its empirical success, theoretical understanding of the superiority of contrastive learning is still limited. In this paper, under linear representation settings, (i) we provably show that contrastive learning outperforms the standard autoencoders and generative adversarial networks, two classical generative unsupervised learning methods, for both feature recovery and in-domain downstream tasks; (ii) we also illustrate the impact of labeled data in supervised contrastive learning. This provides theoretical support for recent findings that contrastive learning with labels improves the performance of learned representations in the in-domain downstream task, but it can harm the performance in transfer learning. We verify our theory with numerical experiments.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 34 of its 67 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

34 of 67 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  7. 2020Supervised Contrastive Learningcited by 2,803
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2019Contrastive Multiview Codingcited by 1,297
22 more
  1. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  2. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  3. 2020A Survey on Contrastive Self-supervised Learningcited by 553
  4. 2017Solving high-dimensional partial differential equations using deep learningcited by 549
  5. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  6. 2016Adversarial Feature Learningcited by 350
  7. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  8. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  9. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  10. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 231
  11. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  12. 2020Provable Meta-Learning of Linear Representationscited by 161
  13. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  14. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  15. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  16. 2021A Broad Study on the Transferability of Visual Representations with Contrastive Learningcited by 78
  17. 2018Triplet-Center Loss for Multi-View 3D Object Retrievalcited by 69
  18. 2016Rate-Optimal Perturbation Bounds for Singular Subspaces with Applications to High-Dimensional Statisticscited by 45
  19. 2018From Principal Subspaces to Principal Components with Linear Autoencoderscited by 38
  20. 2018Heteroskedastic PCA: Algorithm, Optimality, and Applicationscited by 33
  21. 2019A Selective Overview of Deep Learningcited by 31
  22. 2018Spectral Deconfounding via Perturbed Sparse Linear Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 15; 2024: 9; 2025: 11; 2026: 8.

53 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  3. 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 85
  4. 2023Discover and Cure: Concept-aware Mitigation of Spurious Correlationcited by 71
  5. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  6. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  7. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  8. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  9. 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
  10. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  11. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
  12. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
38 more
  1. 2022Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noisecited by 22
  2. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20
  3. 2022Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learningcited by 19
  4. 2023Complexity Matters: Rethinking the Latent Space for Generative Modelingcited by 19
  5. 2024Data-Efficient Contrastive Language-Image Pretraining: Prioritizing Data Quality over Quantitycited by 19
  6. 2023HappyMap: A Generalized Multi-calibration Methodcited by 17
  7. 2023Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networkscited by 17
  8. 2022FIFA: Making Fairness More Generalizable in Classifiers Trained on Imbalanced Datacited by 13
  9. 2024CSA: Data-efficient Mapping of Unimodal Features to Multimodal Featurescited by 10
  10. 2025Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictionscited by 9
  11. 2021Scaffolding Setscited by 8
  12. 2023HyperTime: Hyperparameter Optimization for Combating Temporal Distribution Shiftscited by 8
  13. 2023Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Trainingcited by 7
  14. 2023Unraveling Projection Heads in Contrastive Learning: Insights from Expansion and Shrinkagecited by 6
  15. 2024Variance Alignment Score: A Simple But Tough-to-Beat Data Selection Method for Multimodal Contrastive Learningcited by 6
  16. 2024Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Lossescited by 6
  17. 2024Bootstrap Latents of Nodes and Neighbors for Graph Self-Supervised Learningcited by 6
  18. 2023Deciphering the Projection Head: Representation Evaluation Self-supervised Learningcited by 5
  19. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learningcited by 5
  20. 2023Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shiftcited by 4
  21. 2025PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learningcited by 4
  22. 2022Guided Deep Metric Learningcited by 3
  23. 2022Homomorphic Self-Supervised Learningcited by 3
  24. 2026Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learningcited by 3
  25. 2024Contrastive Learning on Multimodal Analysis of Electronic Health Recordscited by 2
  26. 2025On the Alignment Between Supervised and Self-Supervised Contrastive Learningcited by 2
  27. 2025Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similaritycited by 2
  28. 2025CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encodercited by 2
  29. 2026The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergencecited by 2
  30. 2024When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?cited by 1
  31. 2025DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalizationcited by 1
  32. 2025A Contrastive Learning Foundation Model Based on Perfectly Aligned Sample Pairs for Remote Sensing Imagescited by 1
  33. 2025How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reductioncited by 1
  34. 2025A theoretical framework for self-supervised contrastive learning for continuous dependent datacited by 0
  35. 2025Understanding the Gain from Data Filtering in Multimodal Contrastive Learningcited by 0
  36. 2026Understanding the Role of Rehearsal Scale in Continual Learning under Varying Model Capacitiescited by 0
  37. 2026TRACER: Persistent Regularization for Robust Multimodal Finetuningcited by 0
  38. 2026The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learningcited by 0

50 most cited of 53 All 53 (JSON)