# Knowledge Distillation for BERT Unsupervised Domain Adaptation

arXiv: https://arxiv.org/abs/2010.11478 · HTML: https://mlsanity.com/paper/2010.11478/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [MetaAdapt: Domain Adaptive Few-Shot Misinformation Detection via Meta Learning](https://mlsanity.com/paper/2305.12692/) (2023)
- [DALLMi: Domain Adaption for LLM-based Multi-label Classifier](https://mlsanity.com/paper/2405.01883/) (2024)
- [Explainable Misinformation Detection Across Multiple Social Media Platforms](https://mlsanity.com/paper/2203.11724/) (2022)
- [A Privacy-Preserving Unsupervised Domain Adaptation Framework for Clinical Text Analysis](https://mlsanity.com/paper/2201.07317/) (2022)
- [Dynamically Refined Regularization for Improving Cross-corpora Hate Speech Detection](https://mlsanity.com/paper/2203.12536/) (2022)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [Myers-Briggs personality classification from social media text using pre-trained language models](https://mlsanity.com/paper/2207.04476/) (2022)
- [The economic trade-offs of large language models: A case study](https://mlsanity.com/paper/2306.07402/) (2023)
- [Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data](https://mlsanity.com/paper/2307.03347/) (2023)
- [Zero-shot Cross-lingual Stance Detection via Adversarial Language Adaptation](https://mlsanity.com/paper/2404.14339/) (2024)
- [Adaptive Modality Balanced Online Knowledge Distillation for Brain-Eye-Computer based Dim Object Detection](https://mlsanity.com/paper/2407.01894/) (2024)
- [Domain Classification-based Source-specific Term Penalization for Domain Adaptation in Hate-speech Detection](https://mlsanity.com/paper/2209.08681/) (2022)
- [Similarity-Based Domain Adaptation with LLMs](https://mlsanity.com/paper/2503.05281/) (2025)

## Abstract (arXiv)

A pre-trained language model, BERT, has brought significant performance improvements across a range of natural language processing tasks. Since the model is trained on a large corpus of diverse topics, it shows robust performance for domain shift problems in which data distributions at training (source data) and testing (target data) differ while sharing similarities. Despite its great improvements compared to previous models, it still suffers from performance degradation due to domain shifts. To mitigate such problems, we propose a simple but effective unsupervised domain adaptation method, adversarial adaptation with distillation (AAD), which combines the adversarial discriminative domain adaptation (ADDA) framework with knowledge distillation. We evaluate our approach in the task of cross-domain sentiment classification on 30 domain pairs, advancing the state-of-the-art performance for unsupervised domain adaptation in text sentiment classification.