# Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training

arXiv: https://arxiv.org/abs/2610.20102 · HTML: https://mlsanity.com/paper/2610.20102/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Builds on (most cited first, 16 of 16)

- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge](https://mlsanity.com/paper/2411.19799/) (2024)
- [Balancing Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2004.06748/) (2020)
- [MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2504.02768/) (2025)
- [A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages](https://mlsanity.com/paper/2403.00686/) (2024)
- [BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop](https://mlsanity.com/paper/2602.20092/) (2026)
- [XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs](https://mlsanity.com/paper/2502.19737/) (2025)
- [The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments](https://mlsanity.com/paper/2404.07982/) (2024)
- [Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2204.13869/) (2022)

## Abstract (arXiv)

We investigate two aspects of multilingual pre-training with limited data, namely how language exposure is distributed across English, Dutch, and Chinese, and how that exposure is ordered during training. In particular, we compare balanced multilingual exposure to language-dominant exposure, where one language is exposed with more training data than the others. We also compare mixed training, where multilingual training data is used throughout training, with curriculum training, where monolingual exposure of the dominant language is followed by balanced multilingual exposure. Our results show that multilingual BabyLM performance varies with both the amount and order of language exposure. Mixed training outperforms curriculum learning across all 15 matched configurations, indicating that simultaneous multilingual exposure is more effective than staged exposure in our setting. Several language-dominant mixed training configurations achieve higher scores than the balanced configuration, suggesting that a balanced distribution is not necessarily the highest-performing configuration in our setting. Interestingly, the Dutch-primary configurations achieve the best overall performance.