mlsanityData to 14 Oct 2026

An Augmentation Overlap Theory of Contrastive Learning

arXiv 2511.03114 ↗First posted 5 Nov 2025cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Recently, self-supervised contrastive learning has achieved great success on various tasks. However, its underlying working mechanism is yet unclear. In this paper, we first provide the tightest bounds based on the widely adopted assumption of conditional independence. Further, we relax the conditional independence assumption to a more practical assumption of augmentation overlap and derive the asymptotically closed bounds for the downstream performance. Our proposed augmentation overlap theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations, thus simply aligning the positive samples (augmented views of the same sample) could make contrastive learning cluster intra-class samples together. Moreover, from the newly derived augmentation overlap perspective, we develop an unsupervised metric for the representation evaluation of contrastive learning, which aligns well with the downstream performance almost without relying on additional modules. Code is available at https://github.com/PKU-ML/GARC.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 54 of its 65 references that are in the map. First posted on arXiv 5 Nov 2025.

Builds on

54 of 65 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  12. 2020What Makes for Good Views for Contrastive Learning?cited by 846
38 more
  1. 2020Contrastive Learning with Hard Negative Samplescited by 628
  2. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  3. 2020Understanding the Behaviour of Contrastive Losscited by 398
  4. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  5. 2019On Mutual Information Maximization for Representation Learningcited by 316
  6. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  7. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  8. 2019Revisiting Self-Supervised Visual Representation Learningcited by 251
  9. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  10. 2020Representation Learning via Invariant Causal Mechanismscited by 199
  11. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  12. 2021Decoupled Contrastive Learningcited by 143
  13. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  14. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  15. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 89
  16. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  17. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  18. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 72
  19. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  20. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  21. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  22. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  23. 2022How to Understand Masked Autoencoderscited by 38
  24. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  25. 2021Residual Relaxation for Multi-view Representation Learningcited by 30
  26. 2023On the Generalization of Multi-modal Contrastive Learningcited by 30
  27. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  28. 2021Max-Margin Contrastive Learningcited by 27
  29. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  30. 2023Generalization Analysis for Contrastive Representation Learningcited by 25
  31. 2024Do Generated Data Always Help Contrastive Learning?cited by 25
  32. 2023A Message Passing Perspective on Learning Dynamics of Contrastive Learningcited by 22
  33. 2024Rethinking Graph Masked Autoencoders through Alignment and Uniformitycited by 22
  34. 2023Understanding Contrastive Learning via Distributionally Robust Optimizationcited by 21
  35. 2016Hard Negative Mining for Metric Learning Based Zero-Shot Classificationcited by 18
  36. 2020Functional Regularization for Representation Learning: A Unified Theoretical Perspectivecited by 18
  37. 2023Rethinking Weak Supervision in Helping Contrastive Learningcited by 18
  38. 2025Projection Head is Secretly an Information Bottleneckcited by 11

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026X-VORTEX: Spatio-Temporal Contrastive Learning for Wake Vortex Trajectory Forecastingcited by 0
  2. 2026The geometry of invariant learning: an information-theoretic analysis of data augmentation and generalizationcited by 0
  3. 2026Identifiability of Relational Queries in Multi-View Pretrainingcited by 0
  4. 2026Expressivity In Multimodal Contrastive Learningcited by 0
  5. 2026A Theoretical Framework for Masked Pretraining (MPT)cited by 0