Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation
On the map: Mixture-of-experts → MoE large language models
Abstract (arXiv)
Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additional per-token computation. Given this extra capacity, does adding experts reduce what each additional language costs? To answer this question, we propose a repetition-aware joint scaling law relating number of active parameters, training tokens per language, number of experts, and language coverage. Fitted on 26,300 per-language loss observations from 64 pretrained configurations, our law reduces prediction error on held-out configurations by 28% relative to the strongest evaluated baseline. At matched number of active parameters and training tokens per language, 32 experts reduce the loss penalty of expanding from 12 to 200 languages by 83% compared with a dense model. Our law further shows that broader language coverage favors more experts across a wider range of compute budgets, guiding capacity allocation as multilingual models scale.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 40 of its 51 references that are in the map. First posted on arXiv 14 Oct 2026.
Builds on
40 of 51 references.
Closest ancestors · 5
Same direction, newest first
- 2025Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model14 Jun
- 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts28 May
- 2024LOLA -- An Open-Source Massively Multilingual Large Language Model17 Sep
- 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
Other references · 35
- 2017Attention Is All You Needcited by 60,361
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2020GLU Variants Improve Transformercited by 1,804
- 2019Root Mean Square Layer Normalizationcited by 1,480
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2019Cross-lingual Language Model Pretrainingcited by 1,090
23 more
- 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
- 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
- 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
- 2022Unified Scaling Laws for Routed Language Modelscited by 253
- 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
- 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
- 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
- 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
- 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
- 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
- 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
- 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
- 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
- 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 65
- 2024Scaling Optimal LR Across Token Horizonscited by 40
- 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 38
- 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 31
- 2024Scaling Laws for Multilingual Language Modelscited by 23
- 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
- 2025XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairscited by 12
- 2024Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languagescited by 8
- 2026ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Lawscited by 3
- 2026SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignmentcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
- 2025Towards a Comprehensive Scaling Law of Mixture-of-Experts
- 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
- 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
- 2025Scaling Laws for Upcycling Mixture-of-Experts Language Models
- 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
- 2026How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
- 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts