# Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation

arXiv: https://arxiv.org/abs/2204.00570 · HTML: https://mlsanity.com/paper/2204.00570/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 64 references that are in the map.
- First posted on arXiv 1 Apr 2022.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Community Detection and Stochastic Block Models](https://mlsanity.com/paper/1703.10146/) (2017)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [A DIRT-T Approach to Unsupervised Domain Adaptation](https://mlsanity.com/paper/1802.08735/) (2018)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [BREEDS: Benchmarks for Subpopulation Shift](https://mlsanity.com/paper/2008.04859/) (2020)
- [Understanding Self-Training for Gradual Domain Adaptation](https://mlsanity.com/paper/2002.11361/) (2020)
- [Consistency of spectral clustering in stochastic block models](https://mlsanity.com/paper/1312.2050/) (2013)
- [Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime](https://mlsanity.com/paper/1810.02575/) (2018)
- [Unsupervised Domain Adaptation through Self-Supervision](https://mlsanity.com/paper/1909.11825/) (2019)
- [Extending the WILDS Benchmark for Unsupervised Adaptation](https://mlsanity.com/paper/2112.05090/) (2021)
- [Maximum Density Divergence for Domain Adaptation](https://mlsanity.com/paper/2004.12615/) (2020)
- [Cross-domain Contrastive Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2106.05528/) (2021)

## Built on by (most cited first, 30 of 67)

- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [On the duality between contrastive and non-contrastive self-supervised learning](https://mlsanity.com/paper/2206.02574/) (2022)
- [Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time](https://mlsanity.com/paper/2211.14238/) (2022)
- [Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations](https://mlsanity.com/paper/2204.02683/) (2022)
- [InfoNCE Loss Provably Learns Cluster-Preserving Representations](https://mlsanity.com/paper/2302.07920/) (2023)
- [Contrastive Learning for Unsupervised Domain Adaptation of Time Series](https://mlsanity.com/paper/2206.06243/) (2022)
- [RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness](https://mlsanity.com/paper/2206.14502/) (2022)
- [The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning](https://mlsanity.com/paper/2303.00106/) (2023)
- [ManyDG: Many-domain Generalization for Healthcare Applications](https://mlsanity.com/paper/2301.08834/) (2023)
- [A Graph-Theoretic Framework for Understanding Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2311.03524/) (2023)
- [First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domains](https://mlsanity.com/paper/2211.11719/) (2022)
- [Matrix Information Theory for Self-Supervised Learning](https://mlsanity.com/paper/2305.17326/) (2023)
- [When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysis](https://mlsanity.com/paper/2308.05017/) (2023)
- [Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression](https://mlsanity.com/paper/2306.00788/) (2023)
- [Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysis](https://mlsanity.com/paper/2206.04281/) (2022)
- [SF(DA)2: Source-free Domain Adaptation Through the Lens of Data Augmentation](https://mlsanity.com/paper/2403.10834/) (2024)
- [Rethinking Weak Supervision in Helping Contrastive Learning](https://mlsanity.com/paper/2306.04160/) (2023)
- [Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training](https://mlsanity.com/paper/2312.02914/) (2023)
- [Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension](https://mlsanity.com/paper/2502.05075/) (2025)
- [Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency](https://mlsanity.com/paper/2206.08222/) (2022)
- [Feature contamination: Neural networks learn uncorrelated features and fail to generalize](https://mlsanity.com/paper/2406.03345/) (2024)
- [ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations](https://mlsanity.com/paper/2303.01092/) (2023)
- [MetaAdapt: Domain Adaptive Few-Shot Misinformation Detection via Meta Learning](https://mlsanity.com/paper/2305.12692/) (2023)
- [HyperTime: Hyperparameter Optimization for Combating Temporal Distribution Shifts](https://mlsanity.com/paper/2305.18421/) (2023)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Data](https://mlsanity.com/paper/2311.01420/) (2023)
- [Connect Later: Improving Fine-tuning for Robustness with Targeted Augmentations](https://mlsanity.com/paper/2402.03325/) (2024)
- [Prototypical Partial Optimal Transport for Universal Domain Adaptation](https://mlsanity.com/paper/2408.01089/) (2024)
- [When and How Does In-Distribution Label Help Out-of-Distribution Detection?](https://mlsanity.com/paper/2405.18635/) (2024)
- [Substance or Style: What Does Your Image Embedding Know?](https://mlsanity.com/paper/2307.05610/) (2023)

## Abstract (arXiv)

We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e.g., photographs) and unlabeled data from a target domain (e.g., sketches) are used to learn a classifier for the target domain. Conventional UDA methods (e.g., domain adversarial training) learn domain-invariant features to improve generalization to the target domain. In this paper, we show that contrastive pre-training, which learns features on unlabeled source and target data and then fine-tunes on labeled source data, is competitive with strong UDA methods. However, we find that contrastive pre-training does not learn domain-invariant features, diverging from conventional UDA intuitions. We show theoretically that contrastive pre-training can learn features that vary subtantially across domains but still generalize to the target domain, by disentangling domain and class information. Our results suggest that domain invariance is not necessary for UDA. We empirically validate our theory on benchmark vision datasets.