# An In-Vitro Study on Cross-Lingual Generalization in Language Models

arXiv: https://arxiv.org/abs/2605.26683 · HTML: https://mlsanity.com/paper/2605.26683/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 26 May 2026.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models](https://mlsanity.com/paper/2204.08110/) (2022)
- [Physics of Language Models: Part 1, Learning Hierarchical Language Structures](https://mlsanity.com/paper/2305.13673/) (2023)
- [Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages](https://mlsanity.com/paper/2305.17179/) (2023)
- [Probing the Emergence of Cross-lingual Alignment during LLM Training](https://mlsanity.com/paper/2406.13229/) (2024)
- [Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases](https://mlsanity.com/paper/2502.19249/) (2025)
- [Injecting structural hints: Using language models to study inductive biases in language learning](https://mlsanity.com/paper/2304.13060/) (2023)
- [Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models](https://mlsanity.com/paper/2205.11758/) (2022)
- [MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization](https://mlsanity.com/paper/2407.08818/) (2024)
- [A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank](https://mlsanity.com/paper/2205.04086/) (2022)
- [A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language](https://mlsanity.com/paper/2408.12578/) (2024)
- [One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers](https://mlsanity.com/paper/2506.10766/) (2025)
- [The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models](https://mlsanity.com/paper/2505.14172/) (2025)
- [Beyond Literal Token Overlap: Token Alignability for Multilinguality](https://mlsanity.com/paper/2502.06468/) (2025)
- [Perplexity Cannot Always Tell Right from Wrong](https://mlsanity.com/paper/2601.22950/) (2026)
- [Explaining and Mitigating Crosslingual Tokenizer Inequities](https://mlsanity.com/paper/2510.21909/) (2025)
- [You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models](https://mlsanity.com/paper/2210.07135/) (2022)
- [Measuring Cross-lingual Transfer in Bytes](https://mlsanity.com/paper/2404.08191/) (2024)
- [Creating ConLangs to Probe the Metalinguistic Grammatical Knowledge of LLMs](https://mlsanity.com/paper/2510.07591/) (2025)

## Abstract (arXiv)

Cross-lingual transfer in language models is difficult to study in natural corpora because lexical overlap, morphology, data imbalance, and tokenization are entangled. We introduce an in-vitro framework with two procedurally generated languages that share the same ontology, typed grammar, and compositional structure, but differ in surface realization. This lets us independently vary lexical distance, minority-language proportion, tokenizer training regime, and vocabulary size, while evaluating transfer on a masked minority-language condition whose lexical forms are never observed during training. Across 700 controlled runs, we find that transfer is governed less by tokenizer balance or raw lexical similarity than by whether tokenization preserves reusable cross-lingual substructure. Smaller vocabularies often improve masked transfer by keeping words decomposable into shared fragments, whereas larger vocabularies can turn forms into language-specific atoms. We further show that transfer emerges as a staged process: grammatical and type-level competence precede masked lexical generalization. Finally, we attempt to explain this mechanism through tokenizer bridges and show that bridge strength correlates strongly with masked reachability.