# Knowledge Adaptation: Teaching to Adapt

arXiv: https://arxiv.org/abs/1702.02052 · HTML: https://mlsanity.com/paper/1702.02052/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2017.

## Built on by (most cited first, 17 of 17)

- [Adaptive Multi-Teacher Multi-level Knowledge Distillation](https://mlsanity.com/paper/2103.04062/) (2021)
- [Multi-Source Domain Adaptation with Mixture of Experts](https://mlsanity.com/paper/1809.02256/) (2018)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Your Classifier can Secretly Suffice Multi-Source Domain Adaptation](https://mlsanity.com/paper/2103.11169/) (2021)
- [Student Network Learning via Evolutionary Knowledge Distillation](https://mlsanity.com/paper/2103.13811/) (2021)
- [GoodSAM: Bridging Domain and Capacity Gaps via Segment Anything Model for Distortion-aware Panoramic Semantic Segmentation](https://mlsanity.com/paper/2403.16370/) (2024)
- [M2D2: A Massively Multi-domain Language Modeling Dataset](https://mlsanity.com/paper/2210.07370/) (2022)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation](https://mlsanity.com/paper/2410.01680/) (2024)
- [Knowledge-Adaptation Priors](https://mlsanity.com/paper/2106.08769/) (2021)
- [Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains](https://mlsanity.com/paper/2101.07308/) (2021)
- [GoodSAM++: Bridging Domain and Capacity Gaps via Segment Anything Model for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2408.09115/) (2024)
- [Quantum Transfer Learning for Acceptability Judgements](https://mlsanity.com/paper/2401.07777/) (2024)
- [Human Heterogeneity Invariant Stress Sensing](https://mlsanity.com/paper/2506.02256/) (2025)
- [CARoL: Context-aware Adaptation for Robot Learning](https://mlsanity.com/paper/2506.07006/) (2025)
- [Equally Critical: Samples, Targets, and Their Mappings in Datasets](https://mlsanity.com/paper/2506.01987/) (2025)

## Abstract (arXiv)

Domain adaptation is crucial in many real-world applications where the distribution of the training data differs from the distribution of the test data. Previous Deep Learning-based approaches to domain adaptation need to be trained jointly on source and target domain data and are therefore unappealing in scenarios where models need to be adapted to a large number of domains or where a domain is evolving, e.g. spam detection where attackers continuously change their tactics. To fill this gap, we propose Knowledge Adaptation, an extension of Knowledge Distillation (Bucilua et al., 2006; Hinton et al., 2015) to the domain adaptation scenario. We show how a student model achieves state-of-the-art results on unsupervised domain adaptation from multiple sources on a standard sentiment analysis benchmark by taking into account the domain-specific expertise of multiple teachers and the similarities between their domains. When learning from a single teacher, using domain similarity to gauge trustworthiness is inadequate. To this end, we propose a simple metric that correlates well with the teacher's accuracy in the target domain. We demonstrate that incorporating high-confidence examples selected by this metric enables the student model to achieve state-of-the-art performance in the single-source scenario.