# Black-box Source-free Domain Adaptation via Two-stage Knowledge Distillation

arXiv: https://arxiv.org/abs/2305.07881 · HTML: https://mlsanity.com/paper/2305.07881/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 13 May 2023.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs](https://mlsanity.com/paper/1910.03667/) (2019)
- [Generalized Source-free Domain Adaptation](https://mlsanity.com/paper/2108.01614/) (2021)
- [Source-Free Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2103.16372/) (2021)
- [Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation](https://mlsanity.com/paper/2002.02255/) (2020)
- [Multivariate mixture model for myocardium segmentation combining multi-source images](https://mlsanity.com/paper/1612.08820/) (2016)
- [Test-Time Adaptable Neural Networks for Robust Medical Image Segmentation](https://mlsanity.com/paper/2004.04668/) (2020)
- [Source-Free Domain Adaptive Fundus Image Segmentation with Denoised Pseudo-Labeling](https://mlsanity.com/paper/2109.09735/) (2021)
- [Source-Relaxed Domain Adaptation for Image Segmentation](https://mlsanity.com/paper/2005.03697/) (2020)
- [DINE: Domain Adaptation from Single and Multiple Black-box Predictors](https://mlsanity.com/paper/2104.01539/) (2021)
- [Boundary and Entropy-driven Adversarial Learning for Fundus Image Segmentation](https://mlsanity.com/paper/1906.11143/) (2019)
- [Feature Alignment and Uniformity for Test Time Adaptation](https://mlsanity.com/paper/2303.10902/) (2023)
- [Prototype Knowledge Distillation for Medical Segmentation with Missing Modality](https://mlsanity.com/paper/2303.09830/) (2023)

## Built on by (most cited first, 2 of 2)

- [Semi-supervised Semantic Segmentation via Boosting Uncertainty on Unlabeled Data](https://mlsanity.com/paper/2311.18758/) (2023)
- [Prototypical Distillation and Debiased Tuning for Black-box Unsupervised Domain Adaptation](https://mlsanity.com/paper/2412.20670/) (2024)

## Abstract (arXiv)

Source-free domain adaptation aims to adapt deep neural networks using only pre-trained source models and target data. However, accessing the source model still has a potential concern about leaking the source data, which reveals the patient's privacy. In this paper, we study the challenging but practical problem: black-box source-free domain adaptation where only the outputs of the source model and target data are available. We propose a simple but effective two-stage knowledge distillation method. In Stage uppercaseexpandafterromannumeral1, we train the target model from scratch with soft pseudo-labels generated by the source model in a knowledge distillation manner. In Stage uppercaseexpandafterromannumeral2, we initialize another model as the new student model to avoid the error accumulation caused by noisy pseudo-labels. We feed the images with weak augmentation to the teacher model to guide the learning of the student model. Our method is simple and flexible, and achieves surprising results on three cross-domain segmentation tasks.