mlsanityData to 7 Oct 2026

Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations

arXiv 2204.02683 ↗First posted 6 Apr 2022cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Contrastive learning is a highly effective method for learning representations from unlabeled data. Recent works show that contrastive representations can transfer across domains, leading to simple state-of-the-art algorithms for unsupervised domain adaptation. In particular, a linear classifier trained to separate the representations on the source domain can also predict classes on the target domain accurately, even though the representations of the two domains are far from each other. We refer to this phenomenon as linear transferability. This paper analyzes when and why contrastive representations exhibit linear transferability in a general unsupervised domain adaptation setting. We prove that linear transferability can occur when data from the same class in different domains (e.g., photo dogs and cartoon dogs) are more related with each other than data from different classes in different domains (e.g., photo dogs and cartoon cats) are. Our analyses are in a realistic regime where the source and target domains can have unbounded density ratios and be weakly related, and they have distant representations across domains.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 40 of its 72 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

40 of 72 references.

References · 40

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
28 more
  1. 2021Robust fine-tuning of zero-shot modelscited by 785
  2. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  3. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  4. 2019Bridging Theory and Algorithm for Domain Adaptationcited by 332
  5. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  6. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  7. 2018A DIRT-T Approach to Unsupervised Domain Adaptationcited by 215
  8. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  9. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  10. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  11. 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
  12. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  13. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 85
  14. 2020SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised Domain Adaptationcited by 78
  15. 2021Cross-domain Contrastive Learning for Unsupervised Domain Adaptationcited by 75
  16. 2020Self-training Avoids Using Spurious Features Under Domain Shiftcited by 69
  17. 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 66
  18. 2022A Broad Study of Pre-training for Domain Generalization and Adaptationcited by 64
  19. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  20. 2020In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustnesscited by 49
  21. 2021A Theory of Label Propagation for Subpopulation Shiftcited by 48
  22. 2019On Learning Invariant Representation for Domain Adaptationcited by 38
  23. 2021Contrastive Domain Adaptationcited by 36
  24. 2020PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Modelscited by 33
  25. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  26. 2020Joint Contrastive Learning for Unsupervised Domain Adaptationcited by 25
  27. 2016Neural Structural Correspondence Learning for Domain Adaptationcited by 19
  28. 2013Approximation Algorithm for Sparsest k-Partitioningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 13; 2024: 12; 2025: 8; 2026: 5.

44 papers.

Sort:

By area:

  1. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  2. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  3. 2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 46
  4. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 38
  5. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  6. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 30
  7. 2023ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigmscited by 29
  8. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  9. 2023A Graph-Theoretic Framework for Understanding Open-World Semi-Supervised Learningcited by 22
  10. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  11. 2023When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysiscited by 21
  12. 2022First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domainscited by 20
32 more
  1. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  2. 2024A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classificationcited by 19
  3. 2023Information Flow in Self-Supervised Learningcited by 16
  4. 2022Joint Embedding Self-Supervised Learning in the Kernel Regimecited by 12
  5. 2024Feature contamination: Neural networks learn uncorrelated features and fail to generalizecited by 11
  6. 2023ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representationscited by 10
  7. 2024Connect Later: Improving Fine-tuning for Robustness with Targeted Augmentationscited by 7
  8. 2025Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systemscited by 7
  9. 2023Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysiscited by 6
  10. 2023The Pursuit of Human Labeling: A New Perspective on Unsupervised Learningcited by 6
  11. 2024When and How Does In-Distribution Label Help Out-of-Distribution Detection?cited by 6
  12. 2024Tight PAC-Bayesian Risk Certificates for Contrastive Learningcited by 6
  13. 2023Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shiftcited by 5
  14. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learningcited by 5
  15. 2025PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learningcited by 4
  16. 2024Bridging OOD Detection and Generalization: A Graph-Theoretic Viewcited by 3
  17. 2026Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learningcited by 3
  18. 2023COSTAR: Improved Temporal Counterfactual Estimation with Self-Supervised Learningcited by 2
  19. 2024Understanding Representation Learnability of Nonlinear Self-Supervised Learningcited by 2
  20. 2024Bridging Domains with Approximately Shared Featurescited by 2
  21. 2025SPA++: Generalized Graph Spectral Alignment for Versatile Domain Adaptationcited by 2
  22. 2025On the Alignment Between Supervised and Self-Supervised Contrastive Learningcited by 2
  23. 2023Detecting and Learning Out-of-Distribution Data in the Open world: Algorithm and Theorycited by 1
  24. 2024Adv-SSL: Adversarial Self-Supervised Representation Learning with Theoretical Guaranteescited by 1
  25. 2025FairSSL: Fair Multimodal Self-Supervised Learningcited by 1
  26. 2026Multiview Self-Representation Learning across Heterogeneous Viewscited by 1
  27. 2024Better Representations via Adversarial Training in Pre-Training: A Theoretical Perspectivecited by 0
  28. 2025How to Enhance Downstream Adversarial Robustness (almost) without Touching the Pre-Trained Foundation Model?cited by 0
  29. 2025Beyond the Linear Separability Ceiling: Aligning Representations in VLMscited by 0
  30. 2026LWM-CDE: A Representation Space for Wireless Data Reasoning and Transferabilitycited by 0
  31. 2026TRACER: Persistent Regularization for Robust Multimodal Finetuningcited by 0
  32. 2026Learning a Flow to Self-Supervised Representationscited by 0