mlsanityData to 14 Oct 2026

Investigating the Role of Negatives in Contrastive Representation Learning

arXiv 2106.09943 ↗First posted 18 Jun 2021cs.LG cs.CL stat.MLcited by 54

Not on the map. Why?

Abstract (arXiv)

Noise contrastive learning is a popular technique for unsupervised representation learning. In this approach, a representation is obtained via reduction to supervised learning, where given a notion of semantic similarity, the learner tries to distinguish a similar (positive) example from a collection of random (negative) examples. The success of modern contrastive learning pipelines relies on many parameters such as the choice of data augmentation, the number of negative examples, and the batch size; however, there is limited understanding as to how these parameters interact and affect downstream performance. We focus on disambiguating the role of one of these parameters: the number of negative examples. Theoretically, we show the existence of a collision-coverage trade-off suggesting that the optimal number of negative examples should scale with the number of underlying concepts in the data. Empirically, we scrutinize the role of the number of negatives in both NLP and vision tasks. In the NLP task, we find that the results broadly agree with our theory, while our vision experiments are murkier with performance sometimes even being insensitive to the number of negatives. We discuss plausible explanations for this behavior and suggest future directions to better align theory and practice.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 32 of its 40 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

32 of 40 references.

References · 32

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  8. 2020Supervised Contrastive Learningcited by 2,829
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  11. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  12. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
20 more
  1. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  2. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  3. 2020Debiased Contrastive Learningcited by 356
  4. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  5. 2019On Mutual Information Maximization for Representation Learningcited by 316
  6. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  7. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  8. 2019Contrastive Learning of Structured World Modelscited by 209
  9. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  10. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  11. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  12. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  13. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  14. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  15. 2019A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experimentscited by 58
  16. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  17. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  18. 2020Masked Contrastive Representation Learning for Reinforcement Learningcited by 32
  19. 2020LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understandingcited by 27
  20. 2020Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representationscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 11; 2023: 20; 2024: 5; 2025: 10; 2026: 5.

54 papers.

Sort:

By area:

  1. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  2. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  3. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  4. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  5. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  6. 2022Do More Negative Samples Necessarily Hurt in Contrastive Learning?cited by 39
  7. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  8. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  9. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  10. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 34
  11. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 33
  12. 2022From t-SNE to UMAP with contrastive learningcited by 28
38 more
  1. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  2. 2021Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Towards Better Practicescited by 25
  3. 2023Generalization Analysis for Contrastive Representation Learningcited by 25
  4. 2022Multiple Instance Learning via Iterative Self-Paced Supervised Contrastive Learningcited by 22
  5. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  6. 2023Rethinking Weak Supervision in Helping Contrastive Learningcited by 18
  7. 2023Optimal Sample Complexity of Contrastive Learningcited by 15
  8. 2024Does Negative Sampling Matter? A Review with Insights into its Theory and Applicationscited by 15
  9. 2023Unsupervised Representation Learning for Time Series: A Reviewcited by 14
  10. 2023Identifiable Contrastive Learning with Automatic Feature Importance Discoverycited by 14
  11. 2023ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representationscited by 10
  12. 2023Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimationcited by 10
  13. 2023Detecting Heart Disease from Multi-View Ultrasound Images via Supervised Attention Multiple Instance Learningcited by 7
  14. 2023Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysiscited by 6
  15. 2024Weighted Point Set Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metriccited by 6
  16. 2022ProtoCLIP: Prototypical Contrastive Language Image Pretrainingcited by 5
  17. 2023MSCDA: Multi-level Semantic-guided Contrast Improves Unsupervised Domain Adaptation for Breast MRI Segmentation in Small Datasetscited by 5
  18. 2023Headless Language Models: Learning without Predicting with Contrastive Weight Tyingcited by 5
  19. 2023Certifiably Robust Graph Contrastive Learningcited by 5
  20. 2025A Statistical Theory of Contrastive Learning via Approximate Sufficient Statisticscited by 5
  21. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learningcited by 5
  22. 2025An Augmentation Overlap Theory of Contrastive Learningcited by 5
  23. 2023Towards the Sparseness of Projection Head in Self-Supervised Learningcited by 4
  24. 2025Generalization Analysis for Supervised Contrastive Representation Learning under Non-IID Settingscited by 4
  25. 2023Perfect Alignment May be Poisonous to Graph Contrastive Learningcited by 3
  26. 2025An Augmentation-Aware Theory for Self-Supervised Contrastive Learningcited by 3
  27. 2026Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learningcited by 3
  28. 2023Towards a Rigorous Analysis of Mutual Information in Contrastive Learningcited by 2
  29. 2023Graph Ranking Contrastive Learning: A Extremely Simple yet Efficient Methodcited by 2
  30. 2025Difficult Examples Hurt Unsupervised Contrastive Learning: A Theoretical Perspectivecited by 2
  31. 2025Formal Models of Active Learning from Contrastive Examplescited by 2
  32. 2025On the Alignment Between Supervised and Self-Supervised Contrastive Learningcited by 2
  33. 2022DUEL: Adaptive Duplicate Elimination on Working Memory for Self-Supervised Learningcited by 1
  34. 2024DUEL: Duplicate Elimination on Active Memory for Self-Supervised Class-Imbalanced Learningcited by 1
  35. 2024Adv-SSL: Adversarial Self-Supervised Representation Learning with Theoretical Guaranteescited by 1
  36. 2024On the Generalization and Causal Explanation in Self-Supervised Learningcited by 0
  37. 2025Towards Efficient Contrastive PAC Learningcited by 0
  38. 2025OBSER: Object-Based Sub-Environment Recognition for Zero-Shot Environmental Inferencecited by 0

50 most cited of 54 All 54 (JSON)