Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model
On the map: Mixture-of-experts → MoE large language models
Abstract (arXiv)
The curse of multilinguality phenomenon is a fundamental problem of multilingual Large Language Models (LLMs), where the competition between massive languages results in inferior performance. It mainly comes from limited capacity and negative transfer between dissimilar languages. To address this issue, we propose a method to dynamically group and scale up the parameters of multilingual LLM while boosting positive transfer among similar languages. Specifically, the model is first tuned on monolingual corpus to determine the parameter deviation in each layer and quantify the similarity between languages. Layers with more deviations are extended to mixture-of-experts layers to reduce competition between languages, where one expert module serves one group of similar languages. Experimental results on 18 to 128 languages show that our method reduces the negative transfer between languages and significantly boosts multilingual performance with fewer parameters. Such language group specialization on experts benefits the new language adaptation and reduces the inference on the previous multilingual knowledge learned.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 12 of its 63 references that are in the map. First posted on arXiv 14 Jun 2025.
Builds on
12 of 63 references.
Closest ancestors · 1
Same direction, newest first
Other references · 11
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
- 2023PaLM 2 Technical Reportcited by 1,162
- 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
- 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
- 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
- 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
- 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
- 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
- 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
- 2022Life-long Learning for Multilingual Neural Machine Translation with Knowledge Distillationcited by 5
- 2024Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancementcited by 4
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
- 2026Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit Fusioncited by 0
- 2026A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMDelta Integration into Upcycled MoEcited by 0
- 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognitioncited by 0
- 2026Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocationcited by 0
- 2026Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation14 Oct 2026
- 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition27 Jul 2026
- 2026A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMDelta Integration into Upcycled MoE18 May 2026
- 2026Multilinguality at the Edge: Developing Language Models for the Global South23 Apr 2026
- 2026Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit Fusion20 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts
- 2026Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
- 2026Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
- 2024A Survey on Mixture of Experts in Large Language Models
- 2023Bridging the Language Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs
- 2024Bridging the Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs
- 2024Scaling Laws for Fine-Grained Mixture of Experts
- 2024Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models