# InfoNCE Loss Provably Learns Cluster-Preserving Representations

arXiv: https://arxiv.org/abs/2302.07920 · HTML: https://mlsanity.com/paper/2302.07920/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 56 references that are in the map.
- First posted on arXiv 15 Feb 2023.

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [Predicting What You Already Know Helps: Provable Self-Supervised Learning](https://mlsanity.com/paper/2008.01064/) (2020)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [Contrastive learning, multi-view redundancy, and linear models](https://mlsanity.com/paper/2008.10150/) (2020)

## Built on by (most cited first, 30 of 37)

- [A CLIP-Powered Framework for Robust and Generalizable Data Selection](https://mlsanity.com/paper/2410.11215/) (2024)
- [Memorization in Self-Supervised Learning Improves Downstream Generalization](https://mlsanity.com/paper/2401.12233/) (2024)
- [Revisiting Modularity Maximization for Graph Clustering: A Contrastive Learning Perspective](https://mlsanity.com/paper/2406.14288/) (2024)
- [Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget](https://mlsanity.com/paper/2304.10520/) (2023)
- [Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering](https://mlsanity.com/paper/2307.11030/) (2023)
- [Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse](https://mlsanity.com/paper/2311.05139/) (2023)
- [Understanding Self-Supervised Learning via Gaussian Mixture Models](https://mlsanity.com/paper/2411.03517/) (2024)
- [Tight PAC-Bayesian Risk Certificates for Contrastive Learning](https://mlsanity.com/paper/2412.03486/) (2024)
- [Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning](https://mlsanity.com/paper/2506.04411/) (2025)
- [An Augmentation Overlap Theory of Contrastive Learning](https://mlsanity.com/paper/2511.03114/) (2025)
- [PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning](https://mlsanity.com/paper/2511.12278/) (2025)
- [Taxonomy of the Retrieval System Framework: Pitfalls and Paradigms](https://mlsanity.com/paper/2601.20131/) (2026)
- [Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning](https://mlsanity.com/paper/2603.03530/) (2026)
- [A Theoretical Framework for Preventing Class Collapse in Supervised Contrastive Learning](https://mlsanity.com/paper/2503.08203/) (2025)
- [On the Alignment Between Supervised and Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2510.08852/) (2025)
- [Revisiting Graph Autoencoders as Implicit Contrastive Learners](https://mlsanity.com/paper/2410.10241/) (2024)
- [Graph Diffusion Network for Drug-Gene Prediction](https://mlsanity.com/paper/2502.09335/) (2025)
- [Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning](https://mlsanity.com/paper/2507.12750/) (2025)
- [RankGraph: Unified Heterogeneous Graph Learning for Cross-Domain Recommendation](https://mlsanity.com/paper/2509.02942/) (2025)
- [Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights](https://mlsanity.com/paper/2509.05142/) (2025)
- [Beyond the Academic Monoculture: A Unified Framework and Industrial Perspective for Attributed Graph Clustering](https://mlsanity.com/paper/2603.20829/) (2026)
- [DeepVAT: A Self-Supervised Technique for Cluster Assessment in Image Datasets](https://mlsanity.com/paper/2306.00011/) (2023)
- [Adaptive H&E-IHC information fusion staining framework based on feature extra](https://mlsanity.com/paper/2502.20156/) (2025)
- [BcQLM: Efficient Vision-Language Understanding with Distilled Q-Gated Cross-Modal Fusion](https://mlsanity.com/paper/2509.08715/) (2025)
- [Enriching Knowledge Distillation with Intra-Class Contrastive Learning](https://mlsanity.com/paper/2509.22053/) (2025)
- [Act to See, See to Act: Diffusion-Driven Perception-Action Interplay for Adaptive Policies](https://mlsanity.com/paper/2509.25822/) (2025)
- [Feature Identification for Hierarchical Contrastive Learning](https://mlsanity.com/paper/2510.00837/) (2025)
- [Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projections](https://mlsanity.com/paper/2510.22655/) (2025)
- [Backdoor Attacks on Multi-modal Contrastive Learning](https://mlsanity.com/paper/2601.11006/) (2026)
- [Contrastive Learning for Privacy Enhancements in Industrial Internet of Things](https://mlsanity.com/paper/2602.00515/) (2026)

## Abstract (arXiv)

The goal of contrasting learning is to learn a representation that preserves underlying clusters by keeping samples with similar content, e.g. the ``dogness'' of a dog, close to each other in the space generated by the representation. A common and successful approach for tackling this unsupervised learning problem is minimizing the InfoNCE loss associated with the training samples, where each sample is associated with their augmentations (positive samples such as rotation, crop) and a batch of negative samples (unrelated samples). To the best of our knowledge, it was unanswered if the representation learned by minimizing the InfoNCE loss preserves the underlying data clusters, as it only promotes learning a representation that is faithful to augmentations, i.e., an image and its augmentations have the same representation. Our main result is to show that the representation learned by InfoNCE with a finite number of negative samples is also consistent with respect to clusters in the data, under the condition that the augmentation sets within clusters may be non-overlapping but are close and intertwined, relative to the complexity of the learning function class.