# Semi-supervised Domain Adaptation via Sample-to-Sample Self-Distillation

arXiv: https://arxiv.org/abs/2111.14353 · HTML: https://mlsanity.com/paper/2111.14353/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Semi-supervised Domain Adaptation via Minimax Entropy](https://mlsanity.com/paper/1904.06487/) (2019)
- [Adversarial Domain Adaptation with Domain Mixup](https://mlsanity.com/paper/1912.01805/) (2019)
- [Domain-Specific Batch Normalization for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1906.03950/) (2019)
- [DLOW: Domain Flow for Adaptation and Generalization](https://mlsanity.com/paper/1812.05418/) (2018)
- [Category Anchor-Guided Unsupervised Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/1910.13049/) (2019)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Adversarial Dropout Regularization](https://mlsanity.com/paper/1711.01575/) (2017)
- [Cluster Alignment with a Teacher for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1903.09980/) (2019)
- [Attract, Perturb, and Explore: Learning a Feature Alignment Network for Semi-supervised Domain Adaptation](https://mlsanity.com/paper/2007.09375/) (2020)
- [Explaining Knowledge Distillation by Quantifying the Knowledge](https://mlsanity.com/paper/2003.03622/) (2020)
- [Online Meta-Learning for Multi-Source and Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2004.04398/) (2020)
- [Drop to Adapt: Learning Discriminative Features for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1910.05562/) (2019)

## Built on by (most cited first, 23 of 23)

- [Contrastive Mean-Shift Learning for Generalized Category Discovery](https://mlsanity.com/paper/2404.09451/) (2024)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [ConfMix: Unsupervised Domain Adaptation for Object Detection via Confidence-based Mixing](https://mlsanity.com/paper/2210.11539/) (2022)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies](https://mlsanity.com/paper/2404.09447/) (2024)
- [Combating Label Distribution Shift for Active Domain Adaptation](https://mlsanity.com/paper/2208.06604/) (2022)
- [Learning CNN on ViT: A Hybrid Model to Explicitly Class-specific Boundaries for Domain Adaptation](https://mlsanity.com/paper/2403.18360/) (2024)
- [Bridged-GNN: Knowledge Bridge Learning for Effective Knowledge Transfer](https://mlsanity.com/paper/2308.09499/) (2023)
- [Adaptive Betweenness Clustering for Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2401.11448/) (2024)
- [Enhancing Size Generalization in Graph Neural Networks through Disentangled Representation Learning](https://mlsanity.com/paper/2406.04601/) (2024)
- [Universal Semi-Supervised Domain Adaptation by Mitigating Common-Class Bias](https://mlsanity.com/paper/2403.11234/) (2024)
- [Exploring Few-Shot Adaptation for Activity Recognition on Diverse Domains](https://mlsanity.com/paper/2305.08420/) (2023)
- [Weakly-Supervised Domain Adaptation with Proportion-Constrained Pseudo-Labeling](https://mlsanity.com/paper/2506.22301/) (2025)
- [Decoupling Continual Semantic Segmentation](https://mlsanity.com/paper/2508.05065/) (2025)
- [Pick up the PACE: Fast and Simple Domain Adaptation via Ensemble Pseudo-Labeling](https://mlsanity.com/paper/2205.13508/) (2022)
- [ArcAid: Analysis of Archaeological Artifacts using Drawings](https://mlsanity.com/paper/2211.09480/) (2022)
- [CAusal and collaborative proxy-tasKs lEarning for Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2303.17526/) (2023)
- [Enhancing Visual Domain Adaptation with Source Preparation](https://mlsanity.com/paper/2306.10142/) (2023)
- [GeT: Generative Target Structure Debiasing for Domain Adaptation](https://mlsanity.com/paper/2308.10205/) (2023)
- [Component Adaptive Clustering for Generalized Category Discovery](https://mlsanity.com/paper/2507.01711/) (2025)
- [MoSSDA: A Semi-Supervised Domain Adaptation Framework for Multivariate Time-Series Classification using Momentum Encoder](https://mlsanity.com/paper/2508.08280/) (2025)
- [Domain Adaptation for Ulcerative Colitis Severity Estimation Using Patient-Level Diagnoses](https://mlsanity.com/paper/2509.14573/) (2025)
- [A Multi-View Consistency Framework with Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2601.19266/) (2026)

## Abstract (arXiv)

Semi-supervised domain adaptation (SSDA) is to adapt a learner to a new domain with only a small set of labeled samples when a large labeled dataset is given on a source domain. In this paper, we propose a pair-based SSDA method that adapts a model to the target domain using self-distillation with sample pairs. Each sample pair is composed of a teacher sample from a labeled dataset (i.e., source or labeled target) and its student sample from an unlabeled dataset (i.e., unlabeled target). Our method generates an assistant feature by transferring an intermediate style between the teacher and the student, and then train the model by minimizing the output discrepancy between the student and the assistant. During training, the assistants gradually bridge the discrepancy between the two domains, thus allowing the student to easily learn from the teacher. Experimental evaluation on standard benchmarks shows that our method effectively minimizes both the inter-domain and intra-domain discrepancies, thus achieving significant improvements over recent methods.