mlsanityData to 7 Oct 2026

Difficult Examples Hurt Unsupervised Contrastive Learning: A Theoretical Perspective

arXiv 2501.01317 ↗First posted 2 Jan 2025cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Unsupervised contrastive learning has shown significant performance improvements in recent years, often approaching or even rivaling supervised learning in various tasks. However, its learning mechanism is fundamentally different from supervised learning. Previous works have shown that difficult examples (well-recognized in supervised learning as examples around the decision boundary), which are essential in supervised learning, contribute minimally in unsupervised settings. In this paper, perhaps surprisingly, we find that the direct removal of difficult examples, although reduces the sample size, can boost the downstream classification performance of contrastive learning. To uncover the reasons behind this, we develop a theoretical framework modeling the similarity between different pairs of samples. Guided by this framework, we conduct a thorough theoretical analysis revealing that the presence of difficult examples negatively affects the generalization of contrastive learning. Furthermore, we demonstrate that the removal of these examples, and techniques such as margin tuning and temperature scaling can enhance its generalization bounds, thereby improving performance. Empirically, we propose a simple and efficient mechanism for selecting difficult examples and validate the effectiveness of the aforementioned methods, which substantiates the reliability of our proposed theoretical framework.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 42 of its 50 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

42 of 50 references.

References · 42

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  10. 2020Contrastive Learning with Hard Negative Samplescited by 619
  11. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 544
  12. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
30 more
  1. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  2. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  3. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  4. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  5. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  6. 2018Gradient Harmonized Single-stage Detectorcited by 114
  7. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  8. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  9. 2021Distributional Robustness Loss for Long-tail Learningcited by 84
  10. 2020Sequence-to-Sequence Contrastive Learning for Text Recognitioncited by 70
  11. 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 66
  12. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  13. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  14. 2023Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Datacited by 44
  15. 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
  16. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 32
  17. 2021Residual Relaxation for Multi-view Representation Learningcited by 30
  18. 2023On the Generalization of Multi-modal Contrastive Learningcited by 30
  19. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  20. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  21. 2021Temperature as Uncertainty in Contrastive Learningcited by 25
  22. 2023Contrastive Learning Is Spectral Clustering On Similarity Graphcited by 24
  23. 2023Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Leastcited by 22
  24. 2023A Message Passing Perspective on Learning Dynamics of Contrastive Learningcited by 22
  25. 2023Rethinking Weak Supervision in Helping Contrastive Learningcited by 18
  26. 2024Non-negative Contrastive Learningcited by 18
  27. 2022Toward a Geometrical Understanding of Self-supervised Contrastive Learningcited by 17
  28. 2021Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Frameworkcited by 11
  29. 2021Learning to Discriminate Information for Online Action Detection: Analysis and Applicationcited by 6
  30. 2020Which Strategies Matter for Noisy Label Classification? Insight into Loss and Uncertaintycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Generalization Theory for JEPA-Based World Modelscited by 5
  2. 2026A Theoretical Framework for Masked Pretraining (MPT)cited by 0