mlsanityData to 14 Oct 2026

A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMDelta Integration into Upcycled MoE

arXiv 2605.18083 ↗First posted 18 May 2026cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Expanding Large Language Models~(LLMs) to new languages is a costly endeavor, demanding extensive Continued Pre-Training~(CPT) and data-intensive alignment. While recent data-free merging techniques attempt to bypass alignment by fusing a multilingual CPT-enhanced model with its instruct counterpart, they are plagued by a critical trade-off: mitigating parameter conflicts to preserve original abilities inevitably dilutes new language acquisition, and vice-versa. To resolve this conflict, we introduce method, which upcycles a dense model into a Mixture-of-Experts~(MoE) architecture, allocating different experts to different languages. Alignment ability is then transferred by grafting a MoE-expanded parameter delta~(Deltapost) to the CPT-enhanced base model, bypassing the complex alignment phase. Experiments demonstrate method's superiority even against baselines with similar FLOPs or number of parameters; it improves performance on expanded languages while effectively preserving original capabilities. We further show our approach is highly applicable across different models and Post-training deltas.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 27 of its 36 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

27 of 36 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model14 Jun
  2. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts28 May
  3. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing21 Aug
  4. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translation11 Jul
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 19

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  5. 2024A Survey on Large Language Models for Code Generationcited by 658
  6. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  7. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  8. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  9. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  10. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  11. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  12. 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
7 more
  1. 2024Higher Layers Need More LoRA Expertscited by 102
  2. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  3. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  4. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 56
  5. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  6. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 32
  7. 2025LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjointcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners
  2. 2025The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs
  3. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
  4. 2021MergeDistill: Merging Pre-trained Language Models using Distillation
  5. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
  6. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  7. 2025Model Merging in Pre-training of Large Language Models
  8. 2024Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement