mlsanityData to 7 Oct 2026

Do More Negative Samples Necessarily Hurt in Contrastive Learning?

arXiv 2205.01789 ↗First posted 3 May 2022cs.LG stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

Recent investigations in noise contrastive estimation suggest, both empirically as well as theoretically, that while having more "negative samples" in the contrastive loss improves downstream classification performance initially, beyond a threshold, it hurts downstream performance due to a "collision-coverage" trade-off. But is such a phenomenon inherent in contrastive learning? We show in a simple theoretical setting, where positive pairs are generated by sampling from the underlying latent class (introduced by Saunshi et al. (ICML 2019)), that the downstream performance of the representation optimizing the (population) contrastive loss in fact does not degrade with the number of negative samples. Along the way, we give a structural characterization of the optimal representation in our framework, for noise contrastive estimation. We also provide empirical support for our theoretical results on CIFAR-10 and CIFAR-100 datasets.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 20 of its 38 references that are in the map. First posted on arXiv 3 May 2022.

Builds on

20 of 38 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  9. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  10. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  11. 2017Large Batch Training of Convolutional Networkscited by 529
  12. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
8 more
  1. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  2. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  3. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  4. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  5. 2020Intriguing Properties of Contrastive Lossescited by 129
  6. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  7. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  8. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 11; 2024: 9; 2025: 9; 2026: 6.

39 papers.

Sort:

By area:

  1. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 54
  2. 2022SeeABLE: Soft Discrepancies and Bounded Contrastive Learning for Exposing Deepfakescited by 49
  3. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  4. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  5. 2023Optimal Sample Complexity of Contrastive Learningcited by 15
  6. 2023Unsupervised Representation Learning for Time Series: A Reviewcited by 14
  7. 2024Does Negative Sampling Matter? A Review with Insights into its Theory and Applicationscited by 14
  8. 2023Contrastive Learning with Consistent Representationscited by 6
  9. 2023Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysiscited by 6
  10. 2024Analysis of Using Sigmoid Loss for Contrastive Learningcited by 6
  11. 2023MSCDA: Multi-level Semantic-guided Contrast Improves Unsupervised Domain Adaptation for Breast MRI Segmentation in Small Datasetscited by 5
  12. 2023Headless Language Models: Learning without Predicting with Contrastive Weight Tyingcited by 5
27 more
  1. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learningcited by 5
  2. 2022FG-UAP: Feature-Gathering Universal Adversarial Perturbationcited by 4
  3. 2023Towards the Sparseness of Projection Head in Self-Supervised Learningcited by 4
  4. 2024Table Integration in Data Lakes Unleashed: Pairwise Integrability Judgment, Integrable Set Discovery, and Multi-Tuple Conflict Resolutioncited by 3
  5. 2025Generalization Analysis for Supervised Contrastive Representation Learning under Non-IID Settingscited by 3
  6. 2026Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learningcited by 3
  7. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
  8. 2024PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesiscited by 2
  9. 2025On the Alignment Between Supervised and Self-Supervised Contrastive Learningcited by 2
  10. 2022DUEL: Adaptive Duplicate Elimination on Working Memory for Self-Supervised Learningcited by 1
  11. 2024DUEL: Duplicate Elimination on Active Memory for Self-Supervised Class-Imbalanced Learningcited by 1
  12. 2024Link Prediction with Physics-Inspired Graph Neural Networkscited by 1
  13. 2024Semantic Feature Learning for Universal Unsupervised Cross-Domain Retrievalcited by 1
  14. 2024Adv-SSL: Adversarial Self-Supervised Representation Learning with Theoretical Guaranteescited by 1
  15. 2025JojoSCL: Shrinkage Contrastive Learning for single-cell RNA sequence Clusteringcited by 1
  16. 2025Enhancing Contrastive Learning-based Electrocardiogram Pretrained Model with Patient Memory Queuecited by 1
  17. 2025The Complexity of Finding Local Optima in Contrastive Learningcited by 1
  18. 2023Contrastive Learning for Inference in Dialoguecited by 0
  19. 2023VerA: Versatile Anonymization Applicable to Clinical Facial Photographscited by 0
  20. 2024On the Generalization and Causal Explanation in Self-Supervised Learningcited by 0
  21. 2025Towards Efficient Contrastive PAC Learningcited by 0
  22. 2025OBSER: Object-Based Sub-Environment Recognition for Zero-Shot Environmental Inferencecited by 0
  23. 2026Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrievalcited by 0
  24. 2026SeCo-SBIR: Semantically Consistent Prompt Learning for Zero-Shot Sketch-Based Image Retrievalcited by 0
  25. 2026Expressivity In Multimodal Contrastive Learningcited by 0
  26. 2026Which Tasks Survive Self-Supervised Learning?cited by 0
  27. 2026Optimal VC Dimension of Contrastive Learning with Margincited by 0