# Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model

arXiv: https://arxiv.org/abs/2011.11499 · HTML: https://mlsanity.com/paper/2011.11499/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2020.

## Built on by (most cited first, 24 of 24)

- [Complex Knowledge Base Question Answering: A Survey](https://mlsanity.com/paper/2108.06688/) (2021)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models](https://mlsanity.com/paper/2306.10968/) (2023)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [Maximum Bayes Smatch Ensemble Distillation for AMR Parsing](https://mlsanity.com/paper/2112.07790/) (2021)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [Gender Neutralization for an Inclusive Machine Translation: from Theoretical Foundations to Open Challenges](https://mlsanity.com/paper/2301.10075/) (2023)
- [Cross-domain Activity Recognition via Substructural Optimal Transport](https://mlsanity.com/paper/2102.03353/) (2021)
- [Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Study](https://mlsanity.com/paper/2302.04618/) (2023)
- [Zero-shot Domain Adaptation for Neural Machine Translation with Retrieved Phrase-level Prompts](https://mlsanity.com/paper/2209.11409/) (2022)
- [End-to-End Lexically Constrained Machine Translation for Morphologically Rich Languages](https://mlsanity.com/paper/2106.12398/) (2021)
- [Bilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport](https://mlsanity.com/paper/2210.14378/) (2022)
- [X2Parser: Cross-Lingual and Cross-Domain Framework for Task-Oriented Compositional Semantic Parsing](https://mlsanity.com/paper/2106.03777/) (2021)
- [Input Augmentation Improves Constrained Beam Search for Neural Machine Translation: NTT at WAT 2021](https://mlsanity.com/paper/2106.05450/) (2021)
- [Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings](https://mlsanity.com/paper/2103.06459/) (2021)
- [Improving Lexically Constrained Neural Machine Translation with Source-Conditioned Masked Span Prediction](https://mlsanity.com/paper/2105.05498/) (2021)
- [Enhancing Cross-lingual Prompting with Dual Prompt Augmentation](https://mlsanity.com/paper/2202.07255/) (2022)
- [ZC3: Zero-Shot Cross-Language Code Clone Detection](https://mlsanity.com/paper/2308.13754/) (2023)
- [mPMR: A Multilingual Pre-trained Machine Reader at Scale](https://mlsanity.com/paper/2305.13645/) (2023)
- [Zero-shot Learning of Drug Response Prediction for Preclinical Drug Screening](https://mlsanity.com/paper/2310.12996/) (2023)
- [Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation](https://mlsanity.com/paper/2506.15702/) (2025)
- [Test-Time Adaptation with Perturbation Consistency Learning](https://mlsanity.com/paper/2304.12764/) (2023)
- [A Benchmark for Text Expansion: Datasets, Metrics, and Baselines](https://mlsanity.com/paper/2309.09198/) (2023)
- [Contrastive Learning-based User Identification with Limited Data on Smart Textiles](https://mlsanity.com/paper/2409.07488/) (2024)

## Abstract (arXiv)

Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements over various cross-lingual and low-resource tasks. Through training on one hundred languages and terabytes of texts, cross-lingual language models have proven to be effective in leveraging high-resource languages to enhance low-resource language processing and outperform monolingual models. In this paper, we further investigate the cross-lingual and cross-domain (CLCD) setting when a pretrained cross-lingual language model needs to adapt to new domains. Specifically, we propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features and domain-invariant features from the entangled pretrained cross-lingual representations, given unlabeled raw texts in the source language. Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts. Experimental results show that our proposed method achieves significant performance improvements over the state-of-the-art pretrained cross-lingual language model in the CLCD setting. The source code of this paper is publicly available at https://github.com/lijuntaopku/UFD.