mlsanityData to 14 Oct 2026

Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning

arXiv 2102.06866 ↗First posted 13 Feb 2021cs.LG stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

Instance discriminative self-supervised representation learning has been attracted attention thanks to its unsupervised nature and informative feature representation for downstream tasks. In practice, it commonly uses a larger number of negative samples than the number of supervised classes. However, there is an inconsistency in the existing analysis; theoretically, a large number of negative samples degrade classification performance on a downstream supervised task, while empirically, they improve the performance. We provide a novel framework to analyze this empirical result regarding negative samples using the coupon collector's problem. Our bound can implicitly incorporate the supervised loss of the downstream task in the self-supervised loss by increasing the number of negative samples. We confirm that our proposed analysis holds on real-world benchmark datasets.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 59 of its 97 references that are in the map. First posted on arXiv 13 Feb 2021.

Builds on

59 of 97 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  12. 2017Prototypical Networks for Few-shot Learningcited by 3,015
38 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  7. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  9. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  11. 2016Colorful Image Colorizationcited by 1,114
  12. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  13. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  14. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  15. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  16. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  17. 2018Do Better ImageNet Models Transfer Better?cited by 565
  18. 2019On Variational Bounds of Mutual Informationcited by 559
  19. 2017Large Batch Training of Convolutional Networkscited by 535
  20. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  21. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  22. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  23. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  24. 2015Skip-Thought Vectorscited by 322
  25. 2019On Mutual Information Maximization for Representation Learningcited by 316
  26. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  27. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  28. 2016Learning Representations for Automatic Colorizationcited by 251
  29. 2019Revisiting Self-Supervised Visual Representation Learningcited by 251
  30. 2019Large Scale Adversarial Representation Learningcited by 239
  31. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 232
  32. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  33. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  34. 2020Representation Learning via Invariant Causal Mechanismscited by 199
  35. 2018An efficient framework for learning sentence representationscited by 198
  36. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  37. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 180
  38. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 13; 2023: 13; 2024: 5; 2025: 9; 2026: 3.

47 papers.

Sort:

By area:

  1. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  2. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  3. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  4. 2022Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCocited by 52
  5. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  6. 2022Do More Negative Samples Necessarily Hurt in Contrastive Learning?cited by 39
  7. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  8. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  9. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 34
  10. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 33
  11. 2022From t-SNE to UMAP with contrastive learningcited by 28
  12. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
35 more
  1. 2021Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Towards Better Practicescited by 25
  2. 2023Generalization Analysis for Contrastive Representation Learningcited by 25
  3. 2021Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCEcited by 22
  4. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  5. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  6. 2023Rethinking Weak Supervision in Helping Contrastive Learningcited by 18
  7. 2023Information Flow in Self-Supervised Learningcited by 16
  8. 2024Does Negative Sampling Matter? A Review with Insights into its Theory and Applicationscited by 15
  9. 2022Interventional Contrastive Learning with Meta Semantic Regularizercited by 11
  10. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
  11. 2023A Review of Predictive and Contrastive Self-supervised Learning for Medical Imagescited by 10
  12. 2023Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysiscited by 6
  13. 2023DimCL: Dimensional Contrastive Learning For Improving Self-Supervised Learningcited by 6
  14. 2024Tight PAC-Bayesian Risk Certificates for Contrastive Learningcited by 6
  15. 2022Empirical Evaluation and Theoretical Analysis for Representation Learning: A Surveycited by 5
  16. 2025A Statistical Theory of Contrastive Learning via Approximate Sufficient Statisticscited by 5
  17. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learningcited by 5
  18. 2025An Augmentation Overlap Theory of Contrastive Learningcited by 5
  19. 2025Generalization Analysis for Supervised Contrastive Representation Learning under Non-IID Settingscited by 4
  20. 2022Rethinking Rotation in Self-Supervised Contrastive Learning: Adaptive Positive or Negative Data Augmentationcited by 3
  21. 2025Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventionscited by 3
  22. 2025An Augmentation-Aware Theory for Self-Supervised Contrastive Learningcited by 3
  23. 2026Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learningcited by 3
  24. 2023Towards a Rigorous Analysis of Mutual Information in Contrastive Learningcited by 2
  25. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
  26. 2025Difficult Examples Hurt Unsupervised Contrastive Learning: A Theoretical Perspectivecited by 2
  27. 2025On the Alignment Between Supervised and Self-Supervised Contrastive Learningcited by 2
  28. 2022Ladder Siamese Network: a Method and Insights for Multi-level Self-Supervised Learningcited by 1
  29. 2024When can we Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis?cited by 1
  30. 2024Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learningcited by 1
  31. 2023Contrastive Learning for Inference in Dialoguecited by 0
  32. 2024On the Generalization and Causal Explanation in Self-Supervised Learningcited by 0
  33. 2025GraphMatch: Fusing Language and Graph Representations in a Dynamic Two-Sided Work Marketplacecited by 0
  34. 2026Expressivity In Multimodal Contrastive Learningcited by 0
  35. 2026Which Tasks Survive Self-Supervised Learning?cited by 0