# Joint Progressive Knowledge Distillation and Unsupervised Domain Adaptation

arXiv: https://arxiv.org/abs/2005.07839 · HTML: https://mlsanity.com/paper/2005.07839/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2020.

## Built on by (most cited first, 12 of 12)

- [Factorizing Knowledge in Neural Networks](https://mlsanity.com/paper/2207.03337/) (2022)
- [FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning](https://mlsanity.com/paper/2105.13617/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains](https://mlsanity.com/paper/2101.07308/) (2021)
- [Memory Consistent Unsupervised Off-the-Shelf Model Adaptation for Source-Relaxed Medical Image Segmentation](https://mlsanity.com/paper/2209.07910/) (2022)
- [Rethinking Ensemble-Distillation for Semantic Segmentation Based Unsupervised Domain Adaptation](https://mlsanity.com/paper/2104.14203/) (2021)
- [WIDER & CLOSER: Mixture of Short-channel Distillers for Zero-shot Cross-lingual Named Entity Recognition](https://mlsanity.com/paper/2212.03506/) (2022)
- [Incremental Multi-Target Domain Adaptation for Object Detection with Efficient Domain Transfer](https://mlsanity.com/paper/2104.06476/) (2021)
- [Towards Accurate Cross-Domain In-Bed Human Pose Estimation](https://mlsanity.com/paper/2110.03578/) (2021)
- [Knowledge Distillation for Multi-Target Domain Adaptation in Real-Time Person Re-Identification](https://mlsanity.com/paper/2205.06237/) (2022)
- [3D Reconstruction of Sculptures from Single Images via Unsupervised Domain Adaptation on Implicit Models](https://mlsanity.com/paper/2210.04265/) (2022)
- [Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data](https://mlsanity.com/paper/2307.03347/) (2023)

## Abstract (arXiv)

Currently, the divergence in distributions of design and operational data, and large computational complexity are limiting factors in the adoption of CNNs in real-world applications. For instance, person re-identification systems typically rely on a distributed set of cameras, where each camera has different capture conditions. This can translate to a considerable shift between source (e.g. lab setting) and target (e.g. operational camera) domains. Given the cost of annotating image data captured for fine-tuning in each target domain, unsupervised domain adaptation (UDA) has become a popular approach to adapt CNNs. Moreover, state-of-the-art deep learning models that provide a high level of accuracy often rely on architectures that are too complex for real-time applications. Although several compression and UDA approaches have recently been proposed to overcome these limitations, they do not allow optimizing a CNN to simultaneously address both. In this paper, we propose an unexplored direction -- the joint optimization of CNNs to provide a compressed model that is adapted to perform well for a given target domain. In particular, the proposed approach performs unsupervised knowledge distillation (KD) from a complex teacher model to a compact student model, by leveraging both source and target data. It also improves upon existing UDA techniques by progressively teaching the student about domain-invariant features, instead of directly adapting a compact model on target domain data. Our method is compared against state-of-the-art compression and UDA techniques, using two popular classification datasets for UDA -- Office31 and ImageClef-DA. In both datasets, results indicate that our method can achieve the highest level of accuracy while requiring a comparable or lower time complexity.