mlsanityData to 7 Oct 2026

ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations

arXiv 2303.01092 ↗First posted 2 Mar 2023cs.LG cs.AI cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) is a paradigm that leverages unlabeled data for model training. Empirical studies show that SSL can achieve promising performance in distribution shift scenarios, where the downstream and training distributions differ. However, the theoretical understanding of its transferability remains limited. In this paper, we develop a theoretical framework to analyze the transferability of self-supervised contrastive learning, by investigating the impact of data augmentation on it. Our results reveal that the downstream performance of contrastive learning depends largely on the choice of data augmentation. Moreover, we show that contrastive learning fails to learn domain-invariant features, which limits its transferability. Based on these theoretical insights, we propose a novel method called Augmentation-robust Contrastive Learning (ArCL), which guarantees to learn domain-invariant features and can be easily integrated with existing contrastive learning algorithms. We conduct experiments on several datasets and show that ArCL significantly improves the transferability of contrastive learning.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

42 of 57 references.

References · 42

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2019Invariant Risk Minimizationcited by 1,832
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  11. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  12. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
30 more
  1. 2013Domain Generalization via Invariant Feature Representationcited by 622
  2. 2018Disentangling by Factorisingcited by 614
  3. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  4. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  5. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  6. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  7. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  8. 2019On Mutual Information Maximization for Representation Learningcited by 315
  9. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  10. 2020Domain Generalization using Causal Matchingcited by 224
  11. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  12. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  13. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 182
  14. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  15. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  16. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  17. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
  18. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  19. 2020On Disentangled Representations Learned From Correlated Datacited by 94
  20. 2021Towards a Theoretical Framework of Out-of-Distribution Generalizationcited by 94
  21. 2019Stable Learning via Sample Reweightingcited by 70
  22. 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 66
  23. 2020Stable Prediction with Model Misspecification and Agnostic Distribution Shiftcited by 62
  24. 2020Learning Causal Semantic Representation for Out-of-Distribution Predictioncited by 60
  25. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  26. 2021Self-Supervised Learning Disentangled Group Representation as Featurecited by 47
  27. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  28. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  29. 2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
  30. 2020Enforcing Predictive Invariance across Structured Biomedical Domainscited by 23

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 4.

10 papers.

  1. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 97
  2. 2024GraphCLIP: Enhancing Transferability in Graph Foundation Models for Text-Attributed Graphscited by 58
  3. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  4. 2023Unsupervised Representation Learning for Time Series: A Reviewcited by 14
  5. 2024Adversarial Representation Engineering: A General Model Editing Framework for Large Language Modelscited by 12
  6. 2024Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalizationcited by 12
  7. 2023Task-Oriented Koopman-Based Control with Contrastive Encodercited by 8
  8. 2023MARIO: Model Agnostic Recipe for Improving OOD Generalization of Graph Contrastive Learningcited by 7
  9. 2024Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretrainingcited by 7
  10. 2023Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysiscited by 6