# A Two-Stage Framework with Self-Supervised Distillation For Cross-Domain Text Classification

arXiv: https://arxiv.org/abs/2304.09820 · HTML: https://mlsanity.com/paper/2304.09820/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 18 Apr 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Neural Unsupervised Domain Adaptation in NLP---A Survey](https://mlsanity.com/paper/2006.00632/) (2020)
- [PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models](https://mlsanity.com/paper/2006.09075/) (2020)
- [UDALM: Unsupervised Domain Adaptation through Language Modeling](https://mlsanity.com/paper/2104.07078/) (2021)

## Built on by (most cited first, 3 of 3)

- [TACIT: A Target-Agnostic Feature Disentanglement Framework for Cross-Domain Text Classification](https://mlsanity.com/paper/2312.17263/) (2023)
- [Similarity-Based Domain Adaptation with LLMs](https://mlsanity.com/paper/2503.05281/) (2025)
- [FedSDAF: Leveraging Source Domain Awareness for Enhanced Federated Domain Generalization](https://mlsanity.com/paper/2505.02515/) (2025)

## Abstract (arXiv)

Cross-domain text classification aims to adapt models to a target domain that lacks labeled data. It leverages or reuses rich labeled data from the different but related source domain(s) and unlabeled data from the target domain. To this end, previous work focuses on either extracting domain-invariant features or task-agnostic features, ignoring domain-aware features that may be present in the target domain and could be useful for the downstream task. In this paper, we propose a two-stage framework for cross-domain text classification. In the first stage, we finetune the model with mask language modeling (MLM) and labeled data from the source domain. In the second stage, we further fine-tune the model with self-supervised distillation (SSD) and unlabeled data from the target domain. We evaluate its performance on a public cross-domain text classification benchmark and the experiment results show that our method achieves new state-of-the-art results for both single-source domain adaptations (94.17% uparrow1.03%) and multi-source domain adaptations (95.09% uparrow1.34%).