# A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank

arXiv: https://arxiv.org/abs/2205.04086 · HTML: https://mlsanity.com/paper/2205.04086/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 9 May 2022.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer](https://mlsanity.com/paper/2106.16171/) (2021)
- [Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies](https://mlsanity.com/paper/2202.12312/) (2022)
- [Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach](https://mlsanity.com/paper/2109.04513/) (2021)
- [Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes](https://mlsanity.com/paper/2109.04921/) (2021)

## Built on by (most cited first, 29 of 29)

- [UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining](https://mlsanity.com/paper/2304.09151/) (2023)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [Multilingual Instruction Tuning With Just a Pinch of Multilinguality](https://mlsanity.com/paper/2401.01854/) (2024)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review](https://mlsanity.com/paper/2305.16768/) (2023)
- [MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling](https://mlsanity.com/paper/2403.10691/) (2024)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2502.21228/) (2025)
- [Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training](https://mlsanity.com/paper/2503.18929/) (2025)
- [The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling](https://mlsanity.com/paper/2410.09223/) (2024)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter](https://mlsanity.com/paper/2501.14491/) (2025)
- [An Efficient Approach for Studying Cross-Lingual Transfer in Multilingual Language Models](https://mlsanity.com/paper/2403.20088/) (2024)
- [You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models](https://mlsanity.com/paper/2210.07135/) (2022)
- [GradSim: Gradient-Based Language Grouping for Effective Multilingual Training](https://mlsanity.com/paper/2310.15269/) (2023)
- [Donors and Recipients: On Asymmetric Transfer Across Tasks and Languages with Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2511.13368/) (2025)
- [Analysing Cross-Lingual Transfer in Low-Resourced African Named Entity Recognition](https://mlsanity.com/paper/2309.05311/) (2023)
- [LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs](https://mlsanity.com/paper/2511.14774/) (2025)
- [Language Varieties of Italy: Technology Challenges and Opportunities](https://mlsanity.com/paper/2209.09757/) (2022)
- [Skill Issue: Are Skills Language-Invariant in LLMs?](https://mlsanity.com/paper/2608.25832/) (2026)
- [MaiNLP at SemEval-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatedness](https://mlsanity.com/paper/2404.02570/) (2024)
- [MiniLingua: A Small Open-Source LLM for European Languages](https://mlsanity.com/paper/2512.13298/) (2025)
- [Dialects of Translationese Shape Language Model Learning](https://mlsanity.com/paper/2602.16469/) (2026)
- [Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMs](https://mlsanity.com/paper/2605.01224/) (2026)
- [Why Low-Resource NLP Needs More Than Cross-Lingual Transfer: Lessons Learned from Luxembourgish](https://mlsanity.com/paper/2605.10714/) (2026)
- [An In-Vitro Study on Cross-Lingual Generalization in Language Models](https://mlsanity.com/paper/2605.26683/) (2026)
- [When English Isn't the Best Teacher: Source Language Effects in Cross-Lingual In-Context Learning](https://mlsanity.com/paper/2606.18033/) (2026)
- [Which Languages Transfer Best to Warlpiri? A Similarity-Based Study for Low-Resource ASR](https://mlsanity.com/paper/2607.10256/) (2026)
- [Language Distances are Practical for Equitable Cross-Lingual Transfer](https://mlsanity.com/paper/2609.32331/) (2026)

## Abstract (arXiv)

We show that the choice of pretraining languages affects downstream cross-lingual transfer for BERT-based models. We inspect zero-shot performance in balanced data conditions to mitigate data size confounds, classifying pretraining languages that improve downstream performance as donors, and languages that are improved in zero-shot performance as recipients. We develop a method of quadratic time complexity in the number of languages to estimate these relations, instead of an exponential exhaustive computation of all possible combinations. We find that our method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks. Our findings can inform developers of large-scale multilingual language models in choosing better pretraining configurations.