# Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review

arXiv: https://arxiv.org/abs/2305.16768 · HTML: https://mlsanity.com/paper/2305.16768/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks](https://mlsanity.com/paper/1909.00964/) (2019)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer](https://mlsanity.com/paper/2106.16171/) (2021)
- [When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer](https://mlsanity.com/paper/2110.14782/) (2021)
- [A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank](https://mlsanity.com/paper/2205.04086/) (2022)
- [Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages](https://mlsanity.com/paper/2203.01976/) (2022)
- [Predicting the Performance of Multilingual NLP Models](https://mlsanity.com/paper/2110.08875/) (2021)
- [Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models](https://mlsanity.com/paper/2205.06130/) (2022)
- [Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representations](https://mlsanity.com/paper/1910.05479/) (2019)
- [Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer](https://mlsanity.com/paper/2105.05975/) (2021)
- [Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation](https://mlsanity.com/paper/2209.15236/) (2022)
- [A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT](https://mlsanity.com/paper/2004.09205/) (2020)
- [Transfer Language Selection for Zero-Shot Cross-Lingual Abusive Language Detection](https://mlsanity.com/paper/2206.00962/) (2022)

## Built on by (most cited first, 30 of 50)

- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs](https://mlsanity.com/paper/2407.03963/) (2024)
- [Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models](https://mlsanity.com/paper/2410.01335/) (2024)
- [What Drives Performance in Multilingual Language Models?](https://mlsanity.com/paper/2404.19159/) (2024)
- [Teaching LLMs to Abstain across Languages via Multilingual Feedback](https://mlsanity.com/paper/2406.15948/) (2024)
- [Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance](https://mlsanity.com/paper/2310.10385/) (2023)
- [Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter](https://mlsanity.com/paper/2501.14491/) (2025)
- [Continual Learning Under Language Shift](https://mlsanity.com/paper/2311.01200/) (2023)
- [Multilingual Gradient Word-Order Typology from Universal Dependencies](https://mlsanity.com/paper/2402.01513/) (2024)
- [Leveraging Multilingual Training for Authorship Representation: Enhancing Generalization across Languages and Domains](https://mlsanity.com/paper/2509.16531/) (2025)
- [The Roles of English in Evaluating Multilingual Language Models](https://mlsanity.com/paper/2412.08392/) (2024)
- [We're Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Text](https://mlsanity.com/paper/2404.07304/) (2024)
- [Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement](https://mlsanity.com/paper/2409.06567/) (2024)
- [How Transliterations Improve Crosslingual Alignment](https://mlsanity.com/paper/2409.17326/) (2024)
- [Training Bilingual LMs with Data Constraints in the Targeted Language](https://mlsanity.com/paper/2411.12986/) (2024)
- [Beyond Data Quantity: Key Factors Driving Performance in Multilingual Language Models](https://mlsanity.com/paper/2412.12500/) (2024)
- [Improving Cross-Lingual Phonetic Representation of Low-Resource Languages Through Language Similarity Analysis](https://mlsanity.com/paper/2501.06810/) (2025)
- [Cross-Lingual Transfer of Cultural Knowledge: An Asymmetric Phenomenon](https://mlsanity.com/paper/2506.01675/) (2025)
- [Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models](https://mlsanity.com/paper/2603.29552/) (2026)
- [AAdaM at SemEval-2024 Task 1: Augmentation and Adaptation for Multilingual Semantic Textual Relatedness](https://mlsanity.com/paper/2404.01490/) (2024)
- [Monolingual and Multilingual Misinformation Detection for Low-Resource Languages: A Comprehensive Survey](https://mlsanity.com/paper/2410.18390/) (2024)
- [SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methods](https://mlsanity.com/paper/2505.23714/) (2025)
- [Shared Path: Unraveling Memorization in Multilingual LLMs through Language Similarities](https://mlsanity.com/paper/2505.15722/) (2025)
- [Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation](https://mlsanity.com/paper/2506.03535/) (2025)
- [FilBench: Can LLMs Understand and Generate Filipino?](https://mlsanity.com/paper/2508.03523/) (2025)
- [Model-Based Ranking of Source Languages for Zero-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2510.03202/) (2025)
- [The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model](https://mlsanity.com/paper/2412.07298/) (2024)

## Abstract (arXiv)

In recent years, pre-trained Multilingual Language Models (MLLMs) have shown a strong ability to transfer knowledge across different languages. However, given that the aspiration for such an ability has not been explicitly incorporated in the design of the majority of MLLMs, it is challenging to obtain a unique and straightforward explanation for its emergence. In this review paper, we survey literature that investigates different factors contributing to the capacity of MLLMs to perform zero-shot cross-lingual transfer and subsequently outline and discuss these factors in detail. To enhance the structure of this review and to facilitate consolidation with future studies, we identify five categories of such factors. In addition to providing a summary of empirical evidence from past studies, we identify consensuses among studies with consistent findings and resolve conflicts among contradictory ones. Our work contextualizes and unifies existing research streams which aim at explaining the cross-lingual potential of MLLMs. This review provides, first, an aligned reference point for future research and, second, guidance for a better-informed and more efficient way of leveraging the cross-lingual capacity of MLLMs.