# Unsupervised Multi-Target Domain Adaptation Through Knowledge Distillation

arXiv: https://arxiv.org/abs/2007.07077 · HTML: https://mlsanity.com/paper/2007.07077/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 30 of 47)

- [Advances and Challenges in Meta-Learning: A Technical Review](https://mlsanity.com/paper/2307.04722/) (2023)
- [Factorizing Knowledge in Neural Networks](https://mlsanity.com/paper/2207.03337/) (2022)
- [Balancing Discriminability and Transferability for Source-Free Domain Adaptation](https://mlsanity.com/paper/2206.08009/) (2022)
- [Multi-Target Domain Adaptation with Collaborative Consistency Learning](https://mlsanity.com/paper/2106.03418/) (2021)
- [Curriculum Graph Co-Teaching for Multi-Target Domain Adaptation](https://mlsanity.com/paper/2104.00808/) (2021)
- [A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm](https://mlsanity.com/paper/2310.12244/) (2023)
- [Self-Distillation from the Last Mini-Batch for Consistency Regularization](https://mlsanity.com/paper/2203.16172/) (2022)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Towards Federated Learning against Noisy Labels via Local Self-Regularization](https://mlsanity.com/paper/2208.12807/) (2022)
- [Deep learning for unsupervised domain adaptation in medical imaging: Recent advancements and future perspectives](https://mlsanity.com/paper/2308.01265/) (2023)
- [CoNMix for Source-free Single and Multi-target Domain Adaptation](https://mlsanity.com/paper/2211.03876/) (2022)
- [Domain-Indexing Variational Bayes: Interpretable Domain Index for Domain Adaptation](https://mlsanity.com/paper/2302.02561/) (2023)
- [Unsupervised Domain Adaptation for Semantic Image Segmentation: a Comprehensive Survey](https://mlsanity.com/paper/2112.03241/) (2021)
- [Graph-Relational Domain Adaptation](https://mlsanity.com/paper/2202.03628/) (2022)
- [FedICT: Federated Multi-task Distillation for Multi-access Edge Computing](https://mlsanity.com/paper/2301.00389/) (2023)
- [KD-DLGAN: Data Limited Image Generation via Knowledge Distillation](https://mlsanity.com/paper/2303.17158/) (2023)
- [Domain-Specificity Inducing Transformers for Source-Free Domain Adaptation](https://mlsanity.com/paper/2308.14023/) (2023)
- [Training-Free Model Merging for Multi-target Domain Adaptation](https://mlsanity.com/paper/2407.13771/) (2024)
- [Adapting Fine-Grained Cross-View Localization to Areas without Fine Ground Truth](https://mlsanity.com/paper/2406.00474/) (2024)
- [Rethinking Ensemble-Distillation for Semantic Segmentation Based Unsupervised Domain Adaptation](https://mlsanity.com/paper/2104.14203/) (2021)
- [ADAS: A Direct Adaptation Strategy for Multi-Target Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2203.06811/) (2022)
- [Viewer-Centred Surface Completion for Unsupervised Domain Adaptation in 3D Object Detection](https://mlsanity.com/paper/2209.06407/) (2022)
- [Trust your Good Friends: Source-free Domain Adaptation by Reciprocal Neighborhood Clustering](https://mlsanity.com/paper/2309.00528/) (2023)
- [Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain Transfer](https://mlsanity.com/paper/2104.06476/) (2021)
- [Towards Accurate Cross-Domain In-Bed Human Pose Estimation](https://mlsanity.com/paper/2110.03578/) (2021)
- [Taxonomy-Structured Domain Adaptation](https://mlsanity.com/paper/2306.07874/) (2023)
- [Knowledge Distillation in Federated Edge Learning: A Survey](https://mlsanity.com/paper/2301.05849/) (2023)
- [Aligning Non-Causal Factors for Transformer-Based Source-Free Domain Adaptation](https://mlsanity.com/paper/2311.16294/) (2023)
- [Semantic Segmentation in Multiple Adverse Weather Conditions with Domain Knowledge Retention](https://mlsanity.com/paper/2401.07459/) (2024)
- [A Multi Camera Unsupervised Domain Adaptation Pipeline for Object Detection in Cultural Sites through Adversarial Learning and Self-Training](https://mlsanity.com/paper/2210.00808/) (2022)

## Abstract (arXiv)

Unsupervised domain adaptation (UDA) seeks to alleviate the problem of domain shift between the distribution of unlabeled data from the target domain w.r.t. labeled data from the source domain. While the single-target UDA scenario is well studied in the literature, Multi-Target Domain Adaptation (MTDA) remains largely unexplored despite its practical importance, e.g., in multi-camera video-surveillance applications. The MTDA problem can be addressed by adapting one specialized model per target domain, although this solution is too costly in many real-world applications. Blending multiple targets for MTDA has been proposed, yet this solution may lead to a reduction in model specificity and accuracy. In this paper, we propose a novel unsupervised MTDA approach to train a CNN that can generalize well across multiple target domains. Our Multi-Teacher MTDA (MT-MTDA) method relies on multi-teacher knowledge distillation (KD) to iteratively distill target domain knowledge from multiple teachers to a common student. The KD process is performed in a progressive manner, where the student is trained by each teacher on how to perform UDA for a specific target, instead of directly learning domain adapted features. Finally, instead of combining the knowledge from each teacher, MT-MTDA alternates between teachers that distill knowledge, thereby preserving the specificity of each target (teacher) when learning to adapt to the student. MT-MTDA is compared against state-of-the-art methods on several challenging UDA benchmarks, and empirical results show that our proposed model can provide a considerably higher level of accuracy across multiple target domains. Our code is available at: https://github.com/LIVIAETS/MT-MTDA