mlsanityData to 13 Oct 2026

A Theoretical Analysis of Contrastive Unsupervised Representation Learning

arXiv 1902.09229 ↗First posted 25 Feb 2019cs.LG cs.AI stat.MLcited by 530

Not on the map. Why?

Abstract (arXiv)

Recent empirical works have successfully used unlabeled data to learn feature representations that are broadly useful in downstream classification tasks. Several of these methods are reminiscent of the well-known word2vec embedding algorithm: leveraging availability of pairs of semantically "similar" data points and "negative samples," the learner forces the inner product of representations of similar pairs with each other to be higher on average than with negative samples. The current paper uses the term contrastive learning for such algorithms and presents a theoretical framework for analyzing them by introducing latent classes and hypothesizing that semantically similar points are sampled from the same latent class. This framework allows us to show provable guarantees on the performance of the learned representations on the average classification task that is comprised of a subset of the same set of latent classes. Our generalization bound also shows that learned representations can reduce (labeled) sample complexity on downstream tasks. We conduct controlled experiments in both the text and image domains to support the theory.

Key facts. Cited by 530 papers in the map (data through 13 Oct 2026). First posted on arXiv 25 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2020: 1; 2021: 104; 2022: 120; 2023: 118; 2024: 84; 2025: 60; 2026: 43.

530 papers.

Sort:

By area:

  1. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 567
  2. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 375
  3. 2024The Platonic Representation Hypothesiscited by 369
  4. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  5. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 312
  6. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 309
  7. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 255
  8. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  9. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  10. 2021Co2L: Contrastive Continual Learningcited by 244
  11. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  12. 2021Unsupervised Representation Learning for Time Series with Temporal Neighborhood Codingcited by 223
38 more
  1. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  2. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  3. 2021BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG datacited by 195
  4. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  5. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  6. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 147
  7. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  8. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  9. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  10. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 119
  11. 2022SimMatch: Semi-supervised Learning with Similarity Matchingcited by 118
  12. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 116
  13. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  14. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  15. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  16. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  17. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  18. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  19. 2021Augmentation-Free Self-Supervised Learning on Graphscited by 94
  20. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  21. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  22. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  23. 2021ReSSL: Relational Self-Supervised Learning with Weak Augmentationcited by 83
  24. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  25. 2021Weakly Supervised Contrastive Learningcited by 81
  26. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 79
  27. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 75
  28. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  29. 2021CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representationscited by 72
  30. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  31. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  32. 2022Crafting Better Contrastive Views for Siamese Representation Learningcited by 67
  33. 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 66
  34. 2023Membership Inference Attacks against Synthetic Data through Overfitting Detectioncited by 66
  35. 2021Category Contrast for Unsupervised Domain Adaptation in Visual Taskscited by 65
  36. 2022Audio Self-supervised Learning: A Surveycited by 65
  37. 2021Dissecting Supervised Contrastive Learningcited by 64
  38. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62

50 most cited of 530 All 530 (JSON)