# Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis

arXiv: https://arxiv.org/abs/2304.00395 · HTML: https://mlsanity.com/paper/2304.00395/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 108 references that are in the map.
- First posted on arXiv 1 Apr 2023.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Contrastive learning, multi-view redundancy, and linear models](https://mlsanity.com/paper/2008.10150/) (2020)
- [Decoupled Contrastive Learning](https://mlsanity.com/paper/2110.06848/) (2021)
- [A Primer on PAC-Bayesian Learning](https://mlsanity.com/paper/1901.05353/) (2019)
- [Intriguing Properties of Contrastive Losses](https://mlsanity.com/paper/2011.02803/) (2020)
- [Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning](https://mlsanity.com/paper/2105.15134/) (2021)
- [Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap](https://mlsanity.com/paper/2203.13457/) (2022)
- [Towards the Generalization of Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2111.00743/) (2021)
- [Self-Supervised Learning with Kernel Dependence Maximization](https://mlsanity.com/paper/2106.08320/) (2021)
- [Contrastive estimation reveals topic posterior information to linear models](https://mlsanity.com/paper/2003.02234/) (2020)
- [Investigating the Role of Negatives in Contrastive Representation Learning](https://mlsanity.com/paper/2106.09943/) (2021)
- [CLDA: Contrastive Learning for Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2107.00085/) (2021)

## Built on by (most cited first, 6 of 6)

- [Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses](https://mlsanity.com/paper/2405.18045/) (2024)
- [Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric](https://mlsanity.com/paper/2404.19228/) (2024)
- [Cooperative learning of Pl@ntNet's Artificial Intelligence algorithm: how does it work and how can we improve it?](https://mlsanity.com/paper/2406.03356/) (2024)
- [On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning](https://mlsanity.com/paper/2410.09156/) (2024)
- [On the Similarities of Embeddings in Contrastive Learning](https://mlsanity.com/paper/2506.09781/) (2025)
- [How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction](https://mlsanity.com/paper/2507.11161/) (2025)

## Abstract (arXiv)

Contrastive learning is an efficient approach to self-supervised representation learning. Although recent studies have made progress in the theoretical understanding of contrastive learning, the investigation of how to characterize the clusters of the learned representations is still limited. In this paper, we aim to elucidate the characterization from theoretical perspectives. To this end, we consider a kernel-based contrastive learning framework termed Kernel Contrastive Learning (KCL), where kernel functions play an important role when applying our theoretical results to other frameworks. We introduce a formulation of the similarity structure of learned representations by utilizing a statistical dependency viewpoint. We investigate the theoretical properties of the kernel-based contrastive loss via this formulation. We first prove that the formulation characterizes the structure of representations learned with the kernel-based contrastive learning framework. We show a new upper bound of the classification error of a downstream task, which explains that our theory is consistent with the empirical success of contrastive learning. We also establish a generalization error bound of KCL. Finally, we show a guarantee for the generalization ability of KCL to the downstream classification task via a surrogate bound.