# Knowledge distillation for semi-supervised domain adaptation

arXiv: https://arxiv.org/abs/1908.07355 · HTML: https://mlsanity.com/paper/1908.07355/

## Key facts

- Cited by 12 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 16 Aug 2019.

## Built on by (most cited first, 12 of 12)

- [Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching](https://mlsanity.com/paper/2102.02973/) (2021)
- [FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning](https://mlsanity.com/paper/2105.13617/) (2021)
- [A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance](https://mlsanity.com/paper/2309.12530/) (2023)
- [Transfer Learning in Magnetic Resonance Brain Imaging: a Systematic Review](https://mlsanity.com/paper/2102.01530/) (2021)
- [Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains](https://mlsanity.com/paper/2101.07308/) (2021)
- [Rethinking Ensemble-Distillation for Semantic Segmentation Based Unsupervised Domain Adaptation](https://mlsanity.com/paper/2104.14203/) (2021)
- [Dual-Teacher++: Exploiting Intra-domain and Inter-domain Knowledge with Reliable Transfer for Cardiac Segmentation](https://mlsanity.com/paper/2101.02375/) (2021)
- [Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room](https://mlsanity.com/paper/2108.11801/) (2021)
- [Unsupervised Domain Adaptation via Distilled Discriminative Clustering](https://mlsanity.com/paper/2302.11984/) (2023)
- [Data Augmentation-Based Unsupervised Domain Adaptation In Medical Imaging](https://mlsanity.com/paper/2308.04395/) (2023)
- [WiFi-based Multi-task Sensing](https://mlsanity.com/paper/2111.14619/) (2021)
- [Enhancing Scalability in Recommender Systems through Lottery Ticket Hypothesis and Knowledge Distillation-based Neural Network Pruning](https://mlsanity.com/paper/2401.10484/) (2024)

## Abstract (arXiv)

In the absence of sufficient data variation (e.g., scanner and protocol variability) in annotated data, deep neural networks (DNNs) tend to overfit during training. As a result, their performance is significantly lower on data from unseen sources compared to the performance on data from the same source as the training data. Semi-supervised domain adaptation methods can alleviate this problem by tuning networks to new target domains without the need for annotated data from these domains. Adversarial domain adaptation (ADA) methods are a popular choice that aim to train networks in such a way that the features generated are domain agnostic. However, these methods require careful dataset-specific selection of hyperparameters such as the complexity of the discriminator in order to achieve a reasonable performance. We propose to use knowledge distillation (KD) -- an efficient way of transferring knowledge between different DNNs -- for semi-supervised domain adaption of DNNs. It does not require dataset-specific hyperparameter tuning, making it generally applicable. The proposed method is compared to ADA for segmentation of white matter hyperintensities (WMH) in magnetic resonance imaging (MRI) scans generated by scanners that are not a part of the training set. Compared with both the baseline DNN (trained on source domain only and without any adaption to target domain) and with using ADA for semi-supervised domain adaptation, the proposed method achieves significantly higher WMH dice scores.