# Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer arXiv: https://arxiv.org/abs/2212.01757 ยท HTML: https://mlsanity.com/paper/2212.01757/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 39 references that are in the map. - First posted on arXiv 4 Dec 2022. ## Builds on (most cited first, 22 of 22) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020) - [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019) - [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020) - [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018) - [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020) - [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019) - [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017) - [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020) - [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020) - [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019) - [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019) - [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021) - [First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT](https://mlsanity.com/paper/2101.11109/) (2021) - [How Language-Neutral is Multilingual BERT?](https://mlsanity.com/paper/1911.03310/) (2019) - [Predicting Performance for Natural Language Processing Tasks](https://mlsanity.com/paper/2005.00870/) (2020) - [Probing Multilingual BERT for Genetic and Typological Signals](https://mlsanity.com/paper/2011.02070/) (2020) ## Built on by (most cited first, 11 of 11) - [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023) - [Subobject-level Image Tokenization](https://mlsanity.com/paper/2402.14327/) (2024) - [Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages](https://mlsanity.com/paper/2304.10158/) (2023) - [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026) - [Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter](https://mlsanity.com/paper/2501.14491/) (2025) - [Can Embedding Similarity Predict Cross-Lingual Transfer? A Systematic Study on African Languages](https://mlsanity.com/paper/2601.03168/) (2026) - [X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity](https://mlsanity.com/paper/2310.17166/) (2023) - [NSL-MT: Linguistically Informed Negative Samples for Efficient Machine Translation in Low-Resource Languages](https://mlsanity.com/paper/2511.09537/) (2025) - [Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation](https://mlsanity.com/paper/2608.30541/) (2026) - [Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence](https://mlsanity.com/paper/2609.30535/) (2026) - [Zero-Compute Cross-Lingual Transferability Estimation Using Typological Feature Proxies](https://mlsanity.com/paper/2609.39640/) (2026) ## Abstract (arXiv) Multi-lingual language models (LM), such as mBERT, XLM-R, mT5, mBART, have been remarkably successful in enabling natural language tasks in low-resource languages through cross-lingual transfer from high-resource ones. In this work, we try to better understand how such models, specifically mT5, transfer *any* linguistic and semantic knowledge across languages, even though no explicit cross-lingual signals are provided during pre-training. Rather, only unannotated texts from each language are presented to the model separately and independently of one another, and the model appears to implicitly learn cross-lingual connections. This raises several questions that motivate our study, such as: Are the cross-lingual connections between every language pair equally strong? What properties of source and target language impact the strength of cross-lingual transfer? Can we quantify the impact of those properties on the cross-lingual transfer? In our investigation, we analyze a pre-trained mT5 to discover the attributes of cross-lingual connections learned by the model. Through a statistical interpretation framework over 90 language pairs across three tasks, we show that transfer performance can be modeled by a few linguistic and data-derived features. These observations enable us to interpret cross-lingual understanding of the mT5 model. Through these observations, one can favorably choose the best source language for a task, and can anticipate its training data demands. A key finding of this work is that similarity of syntax, morphology and phonology are good predictors of cross-lingual transfer, significantly more than just the lexical similarity of languages. For a given language, we are able to predict zero-shot performance, that increases on a logarithmic scale with the number of few-shot target language data points.