"UberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset
Not on the map. Why?
Abstract (arXiv)
Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across languages. A further challenge is the performance interference that can arise from joint multilingual training, commonly referred to as the "curse of multilinguality". We study multilingual data curation across thirteen languages and find that many reported regressions are not inherent to multilingual scaling but instead stem from correctable deficiencies in data quality and composition rather than fundamental capacity limits. In controlled bilingual experiments, improving data quality for any single language benefits others: curating English improves non-English performance in 12 of 13 languages, while curating non-English yields reciprocal improvements in English. Bespoke per-language curation produces substantially larger within-language improvements. Extending these findings to large-scale general-purpose training mixtures, we show that curated multilingual allocations comprising under 8% of total tokens remain remarkably effective. We operationalize this approach within an effort that produced a 20T-token pretraining corpus derived entirely from public sources. Models with 3B and 8B parameters trained on a 1T-token random subset achieve competitive multilingual accuracy with 4-10x fewer training FLOPs than strong public baselines, establishing a new Pareto frontier in multilingual performance versus compute. Moreover, these benefits extend to frontier model scale: the 20T-token corpus served as part of the pretraining dataset for Trinity Large (400B/A13B), which exhibits strong multilingual performance relative to its training FLOPs. These results show that targeted, per-language data curation mitigates multilingual interference and enables compute-efficient multilingual scaling.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 40 of its 49 references that are in the map. First posted on arXiv 16 Feb 2026.
Builds on
40 of 49 references.
References · 40
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2025Qwen3 Technical Reportcited by 8,195
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
- 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2019Cross-lingual Language Model Pretrainingcited by 1,088
- 2020Language-agnostic BERT Sentence Embeddingcited by 621
- 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
28 more
- 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
- 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
- 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
- 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
- 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 181
- 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
- 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
- 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
- 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
- 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
- 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
- 2024OLMES: A Standard for Language Model Evaluationscited by 100
- 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
- 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
- 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
- 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
- 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
- 2023Scaling Laws for Multilingual Neural Machine Translationcited by 27
- 2024Scaling Laws for Multilingual Language Modelscited by 22
- 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19
- 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
- 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
- 2023Order Matters in the Presence of Dataset Imbalance for Multilingual Learningcited by 7
- 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluationscited by 7
- 2025Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generationcited by 6
- 2025MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretrainingcited by 4
- 2025Invisible Languages of the LLM Universecited by 4
- 2025Assessing the Role of Data Quality in Training Bilingual Language Modelscited by 2
Built on by
Citing papers per year: 2026: 3.
3 papers.