# Domain Mismatch Doesn't Always Prevent Cross-Lingual Transfer Learning

arXiv: https://arxiv.org/abs/2211.16671 · HTML: https://mlsanity.com/paper/2211.16671/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://mlsanity.com/paper/1805.03620/) (2018)
- [When Does Unsupervised Machine Translation Work?](https://mlsanity.com/paper/2004.05516/) (2020)
- [An Effective Approach to Unsupervised Machine Translation](https://mlsanity.com/paper/1902.01313/) (2019)
- [When and Why is Unsupervised Neural Machine Translation Useless?](https://mlsanity.com/paper/2004.10581/) (2020)
- [Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?](https://mlsanity.com/paper/1909.01638/) (2019)
- [Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework](https://mlsanity.com/paper/1910.04708/) (2019)
- [Morphological Analyzer and Generator for Russian and Ukrainian Languages](https://mlsanity.com/paper/1503.07283/) (2015)
- [Non-Adversarial Unsupervised Word Translation](https://mlsanity.com/paper/1801.06126/) (2018)
- [Unsupervised Bitext Mining and Translation via Self-trained Contextual Embeddings](https://mlsanity.com/paper/2010.07761/) (2020)
- [Learning Crosslingual Word Embeddings without Bilingual Corpora](https://mlsanity.com/paper/1606.09403/) (2016)
- [Unsupervised Neural Machine Translation with SMT as Posterior Regularization](https://mlsanity.com/paper/1901.04112/) (2019)
- [The Source-Target Domain Mismatch Problem in Machine Translation](https://mlsanity.com/paper/1909.13151/) (2019)
- [Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1904.02331/) (2019)
- [Unsupervised Neural Machine Translation Initialized by Unsupervised Statistical Machine Translation](https://mlsanity.com/paper/1810.12703/) (2018)

## Built on by (most cited first, 1 of 1)

- [A survey of neural-network-based methods utilising comparable data for finding translation equivalents](https://mlsanity.com/paper/2410.15144/) (2024)

## Abstract (arXiv)

Cross-lingual transfer learning without labeled target language data or parallel text has been surprisingly effective in zero-shot cross-lingual classification, question answering, unsupervised machine translation, etc. However, some recent publications have claimed that domain mismatch prevents cross-lingual transfer, and their results show that unsupervised bilingual lexicon induction (UBLI) and unsupervised neural machine translation (UNMT) do not work well when the underlying monolingual corpora come from different domains (e.g., French text from Wikipedia but English text from UN proceedings). In this work, we show that a simple initialization regimen can overcome much of the effect of domain mismatch in cross-lingual transfer. We pre-train word and contextual embeddings on the concatenated domain-mismatched corpora, and use these as initializations for three tasks: MUSE UBLI, UN Parallel UNMT, and the SemEval 2017 cross-lingual word similarity task. In all cases, our results challenge the conclusions of prior work by showing that proper initialization can recover a large portion of the losses incurred by domain mismatch.