An Augmentation-Aware Theory for Self-Supervised Contrastive Learning
Not on the map. Why?
Abstract (arXiv)
Self-supervised contrastive learning has emerged as a powerful tool in machine learning and computer vision to learn meaningful representations from unlabeled data. Meanwhile, its empirical success has encouraged many theoretical studies to reveal the learning mechanisms. However, in the existing theoretical research, the role of data augmentation is still under-exploited, especially the effects of specific augmentation types. To fill in the blank, we for the first time propose an augmentation-aware error bound for self-supervised contrastive learning, showing that the supervised risk is bounded not only by the unsupervised risk, but also explicitly by a trade-off induced by data augmentation. Then, under a novel semantic label assumption, we discuss how certain augmentation methods affect the error bound. Lastly, we conduct both pixel- and representation-level experiments to verify our proposed theoretical results.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 29 of its 31 references that are in the map. First posted on arXiv 28 May 2025.
Builds on
29 of 31 references.
References · 29
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2020Exploring Simple Siamese Representation Learningcited by 2,866
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
- 2020What Makes for Good Views for Contrastive Learning?cited by 846
- 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
- 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
- 2021Contrastive Learning Inverts the Data Generating Processcited by 221
17 more
- 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
- 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
- 2020Sequence-to-Sequence Contrastive Learning for Text Recognitioncited by 70
- 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 67
- 2020On Mutual Information in Contrastive Learning for Visual Representationscited by 54
- 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
- 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
- 2023On the Generalization of Multi-modal Contrastive Learningcited by 30
- 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
- 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
- 2023Generalization Analysis for Contrastive Representation Learningcited by 25
- 2023A Message Passing Perspective on Learning Dynamics of Contrastive Learningcited by 22
- 2023Understanding Contrastive Learning via Distributionally Robust Optimizationcited by 21
- 2024Non-negative Contrastive Learningcited by 19
- 2023Rethinking Weak Supervision in Helping Contrastive Learningcited by 18
- 2021Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Frameworkcited by 11
- 2021Learning to Discriminate Information for Online Action Detection: Analysis and Applicationcited by 6
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.