# A Theoretical Analysis of Contrastive Unsupervised Representation Learning

arXiv: https://arxiv.org/abs/1902.09229 · HTML: https://mlsanity.com/paper/1902.09229/

## Key facts

- Cited by 530 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2019.

## Built on by (most cited first, 30 of 530)

- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive](https://mlsanity.com/paper/2402.13228/) (2024)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training](https://mlsanity.com/paper/2101.12699/) (2021)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [Self-supervised Learning is More Robust to Dataset Imbalance](https://mlsanity.com/paper/2110.05025/) (2021)
- [A Kernel-Based View of Language Model Fine-Tuning](https://mlsanity.com/paper/2210.05643/) (2022)
- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2109.09209/) (2021)
- [Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning](https://mlsanity.com/paper/2105.15134/) (2021)
- [SimMatch: Semi-supervised Learning with Similarity Matching](https://mlsanity.com/paper/2203.06915/) (2022)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap](https://mlsanity.com/paper/2203.13457/) (2022)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Graph Foundation Models: Concepts, Opportunities and Challenges](https://mlsanity.com/paper/2310.11829/) (2023)
- [Towards the Generalization of Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2111.00743/) (2021)
- [Understanding Contrastive Learning Requires Incorporating Inductive Biases](https://mlsanity.com/paper/2202.14037/) (2022)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)

## Abstract (arXiv)

Recent empirical works have successfully used unlabeled data to learn feature representations that are broadly useful in downstream classification tasks. Several of these methods are reminiscent of the well-known word2vec embedding algorithm: leveraging availability of pairs of semantically "similar" data points and "negative samples," the learner forces the inner product of representations of similar pairs with each other to be higher on average than with negative samples. The current paper uses the term contrastive learning for such algorithms and presents a theoretical framework for analyzing them by introducing latent classes and hypothesizing that semantically similar points are sampled from the same latent class. This framework allows us to show provable guarantees on the performance of the learned representations on the average classification task that is comprised of a subset of the same set of latent classes. Our generalization bound also shows that learned representations can reduce (labeled) sample complexity on downstream tasks. We conduct controlled experiments in both the text and image domains to support the theory.