# Feature Adaptation of Pre-Trained Language Models across Languages and Domains with Robust Self-Training

arXiv: https://arxiv.org/abs/2009.11538 · HTML: https://mlsanity.com/paper/2009.11538/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2020.

## Built on by (most cited first, 4 of 4)

- [UDALM: Unsupervised Domain Adaptation through Language Modeling](https://mlsanity.com/paper/2104.07078/) (2021)
- [LLM-enhanced Self-training for Cross-domain Constituency Parsing](https://mlsanity.com/paper/2311.02660/) (2023)
- [Cross-domain Chinese Sentence Pattern Parsing](https://mlsanity.com/paper/2402.16311/) (2024)
- [Learning to Generalize Unseen Domains via Multi-Source Meta Learning for Text Classification](https://mlsanity.com/paper/2409.13787/) (2024)

## Abstract (arXiv)

Adapting pre-trained language models (PrLMs) (e.g., BERT) to new domains has gained much attention recently. Instead of fine-tuning PrLMs as done in most previous work, we investigate how to adapt the features of PrLMs to new domains without fine-tuning. We explore unsupervised domain adaptation (UDA) in this paper. With the features from PrLMs, we adapt the models trained with labeled data from the source domain to the unlabeled target domain. Self-training is widely used for UDA which predicts pseudo labels on the target domain data for training. However, the predicted pseudo labels inevitably include noise, which will negatively affect training a robust model. To improve the robustness of self-training, in this paper we present class-aware feature self-distillation (CFd) to learn discriminative features from PrLMs, in which PrLM features are self-distilled into a feature adaptation module and the features from the same class are more tightly clustered. We further extend CFd to a cross-language setting, in which language discrepancy is studied. Experiments on two monolingual and multilingual Amazon review datasets show that CFd can consistently improve the performance of self-training in cross-domain and cross-language settings.