# Rethinking Weak Supervision in Helping Contrastive Learning

arXiv: https://arxiv.org/abs/2306.04160 · HTML: https://mlsanity.com/paper/2306.04160/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 30 of 35)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [NLNL: Negative Learning for Noisy Labels](https://mlsanity.com/paper/1908.07387/) (2019)
- [Selective-Supervised Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2203.04181/) (2022)
- [Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap](https://mlsanity.com/paper/2203.13457/) (2022)
- [Towards the Generalization of Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2111.00743/) (2021)
- [Multi-Objective Interpolation Training for Robustness to Label Noise](https://mlsanity.com/paper/2012.04462/) (2020)
- [Jo-SRC: A Contrastive Approach for Combating Noisy Labels](https://mlsanity.com/paper/2103.13029/) (2021)
- [A Broad Study on the Transferability of Visual Representations with Contrastive Learning](https://mlsanity.com/paper/2103.13517/) (2021)
- [Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2204.00570/) (2022)
- [Class-Aware Contrastive Semi-Supervised Learning](https://mlsanity.com/paper/2203.02261/) (2022)
- [Investigating the Role of Negatives in Contrastive Representation Learning](https://mlsanity.com/paper/2106.09943/) (2021)
- [Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning](https://mlsanity.com/paper/2102.06866/) (2021)
- [Investigating Why Contrastive Learning Benefits Robustness Against Label Noise](https://mlsanity.com/paper/2201.12498/) (2022)
- [Robust Contrastive Learning against Noisy Views](https://mlsanity.com/paper/2201.04309/) (2022)
- [Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learning](https://mlsanity.com/paper/2204.07596/) (2022)
- [Contrastive Learning Improves Model Robustness Under Label Noise](https://mlsanity.com/paper/2104.08984/) (2021)
- [SelfMatch: Combining Contrastive Self-Supervision and Consistency for Semi-Supervised Learning](https://mlsanity.com/paper/2101.06480/) (2021)
- [Joint Negative and Positive Learning for Noisy Labels](https://mlsanity.com/paper/2104.06574/) (2021)
- [Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions](https://mlsanity.com/paper/2210.01883/) (2022)
- [Residual Relaxation for Multi-view Representation Learning](https://mlsanity.com/paper/2110.15348/) (2021)
- [Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding](https://mlsanity.com/paper/2205.14814/) (2022)
- [Semi-supervised Contrastive Learning with Similarity Co-calibration](https://mlsanity.com/paper/2105.07387/) (2021)
- [On the Surrogate Gap between Contrastive and Supervised Losses](https://mlsanity.com/paper/2110.02501/) (2021)
- [Contrastive Regularization for Semi-Supervised Learning](https://mlsanity.com/paper/2201.06247/) (2022)

## Built on by (most cited first, 18 of 18)

- [Do Generated Data Always Help Contrastive Learning?](https://mlsanity.com/paper/2403.12448/) (2024)
- [Non-negative Contrastive Learning](https://mlsanity.com/paper/2403.12459/) (2024)
- [Adversarial Examples Are Not Real Features](https://mlsanity.com/paper/2310.18936/) (2023)
- [Identifiable Contrastive Learning with Automatic Feature Importance Discovery](https://mlsanity.com/paper/2310.18904/) (2023)
- [Projection Head is Secretly an Information Bottleneck](https://mlsanity.com/paper/2503.00507/) (2025)
- [Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learning](https://mlsanity.com/paper/2311.02687/) (2023)
- [On the Role of Discrete Tokenization in Visual Representation Learning](https://mlsanity.com/paper/2407.09087/) (2024)
- [An Augmentation Overlap Theory of Contrastive Learning](https://mlsanity.com/paper/2511.03114/) (2025)
- [A Generalization Theory for JEPA-Based World Models](https://mlsanity.com/paper/2606.27014/) (2026)
- [An Augmentation-Aware Theory for Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2505.22196/) (2025)
- [An Inclusive Theoretical Framework of Robust Supervised Contrastive Loss against Label Noise](https://mlsanity.com/paper/2501.01130/) (2025)
- [Difficult Examples Hurt Unsupervised Contrastive Learning: A Theoretical Perspective](https://mlsanity.com/paper/2501.01317/) (2025)
- [Weakly-Supervised Contrastive Learning for Imprecise Class Labels](https://mlsanity.com/paper/2505.22028/) (2025)
- [JojoSCL: Shrinkage Contrastive Learning for single-cell RNA sequence Clustering](https://mlsanity.com/paper/2506.00410/) (2025)
- [Generalizing Supervised Contrastive learning: A Projection Perspective](https://mlsanity.com/paper/2506.09810/) (2025)
- [How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction](https://mlsanity.com/paper/2507.11161/) (2025)
- [Weak Supervision for Real World Graphs](https://mlsanity.com/paper/2506.02451/) (2025)
- [A Theoretical Framework for Masked Pretraining (MPT)](https://mlsanity.com/paper/2609.06460/) (2026)

## Abstract (arXiv)

Contrastive learning has shown outstanding performances in both supervised and unsupervised learning, and has recently been introduced to solve weakly supervised learning problems such as semi-supervised learning and noisy label learning. Despite the empirical evidence showing that semi-supervised labels improve the representations of contrastive learning, it remains unknown if noisy supervised information can be directly used in training instead of after manual denoising. Therefore, to explore the mechanical differences between semi-supervised and noisy-labeled information in helping contrastive learning, we establish a unified theoretical framework of contrastive learning under weak supervision. Specifically, we investigate the most intuitive paradigm of jointly training supervised and unsupervised contrastive losses. By translating the weakly supervised information into a similarity graph under the framework of spectral clustering based on the posterior probability of weak labels, we establish the downstream classification error bound. We prove that semi-supervised labels improve the downstream error bound whereas noisy labels have limited effects under such a paradigm. Our theoretical findings here provide new insights for the community to rethink the role of weak supervision in helping contrastive learning.