# A Call for More Rigor in Unsupervised Cross-lingual Learning

arXiv: https://arxiv.org/abs/2004.14958 · HTML: https://mlsanity.com/paper/2004.14958/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 30 of 37)

- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer](https://mlsanity.com/paper/2305.14857/) (2023)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings](https://mlsanity.com/paper/2211.06127/) (2022)
- [Predicting the Performance of Multilingual NLP Models](https://mlsanity.com/paper/2110.08875/) (2021)
- [Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution](https://mlsanity.com/paper/2305.17325/) (2023)
- [MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning](https://mlsanity.com/paper/2104.07908/) (2021)
- [PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining](https://mlsanity.com/paper/2108.01887/) (2021)
- [Improving Word Translation via Two-Stage Contrastive Learning](https://mlsanity.com/paper/2203.08307/) (2022)
- [Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment](https://mlsanity.com/paper/2406.19759/) (2024)
- [Kompetencer: Fine-grained Skill Classification in Danish Job Postings via Distant Supervision and Transfer Learning](https://mlsanity.com/paper/2205.01381/) (2022)
- [Detecting ChatGPT: A Survey of the State of Detecting ChatGPT-Generated Text](https://mlsanity.com/paper/2309.07689/) (2023)
- [On the Limitations of Simulating Active Learning](https://mlsanity.com/paper/2305.13342/) (2023)
- [Wine is Not v i n. -- On the Compatibility of Tokenizations Across Languages](https://mlsanity.com/paper/2109.05772/) (2021)
- [So Different Yet So Alike! Constrained Unsupervised Text Style Transfer](https://mlsanity.com/paper/2205.04093/) (2022)
- [Improving Bilingual Lexicon Induction with Cross-Encoder Reranking](https://mlsanity.com/paper/2210.16953/) (2022)
- [The Roles of English in Evaluating Multilingual Language Models](https://mlsanity.com/paper/2412.08392/) (2024)
- [Bilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport](https://mlsanity.com/paper/2210.14378/) (2022)
- [Majority Voting with Bidirectional Pre-translation For Bitext Retrieval](https://mlsanity.com/paper/2103.06369/) (2021)
- [Average Is Not Enough: Caveats of Multilingual Evaluation](https://mlsanity.com/paper/2301.01269/) (2023)
- [Leveraging Auxiliary Domain Parallel Data in Intermediate Task Fine-tuning for Low-resource Translation](https://mlsanity.com/paper/2306.01382/) (2023)
- [Translating away Translationese without Parallel Data](https://mlsanity.com/paper/2310.18830/) (2023)
- [Flow-Adapter Architecture for Unsupervised Machine Translation](https://mlsanity.com/paper/2204.12225/) (2022)
- [Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval](https://mlsanity.com/paper/2302.01626/) (2023)
- [Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impact](https://mlsanity.com/paper/2410.17532/) (2024)
- [LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them?](https://mlsanity.com/paper/2601.20009/) (2026)
- [Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Models](https://mlsanity.com/paper/2212.07752/) (2022)
- [ProMap: Effective Bilingual Lexicon Induction via Language Model Prompting](https://mlsanity.com/paper/2310.18778/) (2023)
- [LangSAMP: Language-Script Aware Multilingual Pretraining](https://mlsanity.com/paper/2409.18199/) (2024)
- [Self-Learning for Zero Shot Neural Machine Translation](https://mlsanity.com/paper/2103.05951/) (2021)

## Abstract (arXiv)

We review motivations, definition, approaches, and methodology for unsupervised cross-lingual learning and call for a more rigorous position in each of them. An existing rationale for such research is based on the lack of parallel data for many of the world's languages. However, we argue that a scenario without any parallel data and abundant monolingual data is unrealistic in practice. We also discuss different training signals that have been used in previous work, which depart from the pure unsupervised setting. We then describe common methodological issues in tuning and evaluation of unsupervised cross-lingual models and present best practices. Finally, we provide a unified outlook for different types of research in this area (i.e., cross-lingual word embeddings, deep multilingual pretraining, and unsupervised machine translation) and argue for comparable evaluation of these models.