# Surprisingly Simple Semi-Supervised Domain Adaptation with Pretraining and Consistency

arXiv: https://arxiv.org/abs/2101.12727 · HTML: https://mlsanity.com/paper/2101.12727/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 29 Jan 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [VisDA: The Visual Domain Adaptation Challenge](https://mlsanity.com/paper/1710.06924/) (2017)
- [Semi-supervised Domain Adaptation via Minimax Entropy](https://mlsanity.com/paper/1904.06487/) (2019)
- [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019)
- [Interpolation Consistency Training for Semi-Supervised Learning](https://mlsanity.com/paper/1903.03825/) (2019)
- [A DIRT-T Approach to Unsupervised Domain Adaptation](https://mlsanity.com/paper/1802.08735/) (2018)
- [Universal Domain Adaptation through Self Supervision](https://mlsanity.com/paper/2002.07953/) (2020)
- [Unsupervised Domain Adaptation through Self-Supervision](https://mlsanity.com/paper/1909.11825/) (2019)
- [Adversarial Dropout Regularization](https://mlsanity.com/paper/1711.01575/) (2017)
- [Domain-Adversarial Neural Networks](https://mlsanity.com/paper/1412.4446/) (2014)
- [Attract, Perturb, and Explore: Learning a Feature Alignment Network for Semi-supervised Domain Adaptation](https://mlsanity.com/paper/2007.09375/) (2020)
- [Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation](https://mlsanity.com/paper/2010.04647/) (2020)

## Built on by (most cited first, 19 of 19)

- [Extending the WILDS Benchmark for Unsupervised Adaptation](https://mlsanity.com/paper/2112.05090/) (2021)
- [Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2204.00570/) (2022)
- [CLDA: Contrastive Learning for Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2107.00085/) (2021)
- [Test time Adaptation through Perturbation Robustness](https://mlsanity.com/paper/2110.10232/) (2021)
- [Concurrent Subsidiary Supervision for Unsupervised Source-Free Domain Adaptation](https://mlsanity.com/paper/2207.13247/) (2022)
- [Subsidiary Prototype Alignment for Universal Domain Adaptation](https://mlsanity.com/paper/2210.15909/) (2022)
- [Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training](https://mlsanity.com/paper/2312.02914/) (2023)
- [Partial Domain Adaptation without Domain Alignment](https://mlsanity.com/paper/2108.12867/) (2021)
- [Adaptive Betweenness Clustering for Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2401.11448/) (2024)
- [Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shift](https://mlsanity.com/paper/2312.03318/) (2023)
- [Uncertainty-Guided Mixup for Semi-Supervised Domain Adaptation without Source Data](https://mlsanity.com/paper/2107.06707/) (2021)
- [Low-confidence Samples Matter for Domain Adaptation](https://mlsanity.com/paper/2202.02802/) (2022)
- [Universal Semi-Supervised Domain Adaptation by Mitigating Common-Class Bias](https://mlsanity.com/paper/2403.11234/) (2024)
- [Representation Transfer Learning via Multiple Pre-trained models for Linear Regression](https://mlsanity.com/paper/2305.16440/) (2023)
- [Prototypical Cross-domain Knowledge Transfer for Cervical Dysplasia Visual Inspection](https://mlsanity.com/paper/2308.09983/) (2023)
- [Domain Adaptation and Active Learning for Fine-Grained Recognition in the Field of Biodiversity](https://mlsanity.com/paper/2110.11778/) (2021)
- [Pick up the PACE: Fast and Simple Domain Adaptation via Ensemble Pseudo-Labeling](https://mlsanity.com/paper/2205.13508/) (2022)
- [Towards Realizing the Value of Labeled Target Samples: a Two-Stage Approach for Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2304.10762/) (2023)
- [PseudoCal: A Source-Free Approach to Unsupervised Uncertainty Calibration in Domain Adaptation](https://mlsanity.com/paper/2307.07489/) (2023)

## Abstract (arXiv)

Most modern unsupervised domain adaptation (UDA) approaches are rooted in domain alignment, i.e., learning to align source and target features to learn a target domain classifier using source labels. In semi-supervised domain adaptation (SSDA), when the learner can access few target domain labels, prior approaches have followed UDA theory to use domain alignment for learning. We show that the case of SSDA is different and a good target classifier can be learned without needing alignment. We use self-supervised pretraining (via rotation prediction) and consistency regularization to achieve well separated target clusters, aiding in learning a low error target classifier. With our Pretraining and Consistency (PAC) approach, we achieve state of the art target accuracy on this semi-supervised domain adaptation task, surpassing multiple adversarial domain alignment methods, across multiple datasets. PAC, while using simple techniques, performs remarkably well on large and challenging SSDA benchmarks like DomainNet and Visda-17, often outperforming recent state of the art by sizeable margins. Code for our experiments can be found at https://github.com/venkatesh-saligrama/PAC