mlsanityData to 14 Oct 2026

Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap

arXiv 2203.13457 ↗First posted 25 Mar 2022cs.LG cs.CV stat.MLcited by 105

Not on the map. Why?

Abstract (arXiv)

Recently, contrastive learning has risen to be a promising approach for large-scale self-supervised learning. However, theoretical understanding of how it works is still unclear. In this paper, we propose a new guarantee on the downstream performance without resorting to the conditional independence assumption that is widely adopted in previous work but hardly holds in practice. Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations, thus simply aligning the positive samples (augmented views of the same sample) could make contrastive learning cluster intra-class samples together. Based on this augmentation overlap perspective, theoretically, we obtain asymptotically closed bounds for downstream performance under weaker assumptions, and empirically, we propose an unsupervised model selection metric ARC that aligns well with downstream accuracy. Our theory suggests an alternative understanding of contrastive learning: the role of aligning positive samples is more like a surrogate task than an ultimate goal, and the overlapped augmented views (i.e., the chaos) create a ladder for contrastive learning to gradually learn class-separated representations. The code for computing ARC is available at https://github.com/zhangq327/ARC.

Key facts. Cited by 105 papers in the map (data through 14 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 25 Mar 2022.

Builds on

24 of 33 references.

References · 24

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  10. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  11. 2019On Variational Bounds of Mutual Informationcited by 559
  12. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
12 more
  1. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  2. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  3. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  4. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  5. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  6. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  7. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  8. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  9. 2014Scale-Invariant Convolutional Neural Networkscited by 40
  10. 2020SelfAugment: Automatic Augmentation Policies for Self-Supervised Learningcited by 32
  11. 2021Residual Relaxation for Multi-view Representation Learningcited by 30
  12. 2020About contrastive unsupervised representation learning for classification and its convergencecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 37; 2024: 25; 2025: 19; 2026: 10.

105 papers.

Sort:

By area:

  1. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  2. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  3. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  4. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  5. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 40
  6. 2023Simple and Asymmetric Graph Contrastive Learning without Augmentationscited by 38
  7. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  8. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 34
  9. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 33
  10. 2022An Embarrassingly Simple Backdoor Attack on Self-supervised Learningcited by 31
  11. 2024Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuningcited by 31
  12. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 31
38 more
  1. 2023On the Generalization of Multi-modal Contrastive Learningcited by 30
  2. 2023Generalization Analysis for Contrastive Representation Learningcited by 25
  3. 2023Contrastive Learning Is Spectral Clustering On Similarity Graphcited by 25
  4. 2024Do Generated Data Always Help Contrastive Learning?cited by 25
  5. 2023Spatiotemporal Self-supervised Learning for Point Clouds in the Wildcited by 23
  6. 2023A Message Passing Perspective on Learning Dynamics of Contrastive Learningcited by 22
  7. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  8. 2023Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learningcited by 20
  9. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  10. 2023On the Effectiveness of Out-of-Distribution Data in Self-Supervised Long-Tail Learningcited by 20
  11. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  12. 2024Non-negative Contrastive Learningcited by 19
  13. 2023Rethinking Weak Supervision in Helping Contrastive Learningcited by 18
  14. 2023Finding Order in Chaos: A Novel Data Augmentation Method for Time Series in Contrastive Learningcited by 18
  15. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  16. 2024SINDER: Repairing the Singular Defects of DINOv2cited by 18
  17. 2023Information Flow in Self-Supervised Learningcited by 16
  18. 2023Adversarial Examples Are Not Real Featurescited by 16
  19. 2023Unsupervised Representation Learning for Time Series: A Reviewcited by 14
  20. 2023Identifiable Contrastive Learning with Automatic Feature Importance Discoverycited by 14
  21. 2023Rethinking Dimensional Rationale in Graph Contrastive Learning from Causal Perspectivecited by 13
  22. 2023Towards Distribution-Agnostic Generalized Category Discoverycited by 12
  23. 2022Integrating Prior Knowledge in Contrastive Learning with Kernelcited by 11
  24. 2023Combating Representation Learning Disparity with Geometric Harmonizationcited by 11
  25. 2025Projection Head is Secretly an Information Bottleneckcited by 11
  26. 2022Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learningcited by 9
  27. 2023On the Difficulty of Defending Contrastive Learning against Backdoor Attackscited by 9
  28. 2024mathbbX-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphscited by 9
  29. 2024Localizing Memorization in SSL Vision Encoderscited by 9
  30. 2022MetaMask: Revisiting Dimensional Confounder for Self-Supervised Learningcited by 7
  31. 2023Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learningcited by 7
  32. 2024Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretrainingcited by 7
  33. 2024On the Role of Discrete Tokenization in Visual Representation Learningcited by 7
  34. 2024Understanding the Role of Equivariance in Self-supervised Learningcited by 7
  35. 2023Evaluating Self-Supervised Learning via Risk Decompositioncited by 6
  36. 2023Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysiscited by 6
  37. 2023CAME: Contrastive Automated Model Evaluationcited by 6
  38. 2023Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapsecited by 6

50 most cited of 105 All 105 (JSON)