mlsanityData to 14 Oct 2026

MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts

arXiv 2508.19268 ↗First posted 22 Aug 2025cs.CL cs.AIcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Despite LLMs' excellent code creation capabilities, multilingual code generation remains extremely challenging. To address this, we intent to improve the multi-programming-lingual (MultiPL) performance of the base LLMs while retaining the most popular ones using restricted computational resources. We consider MultiPL to be a special case of multiple natural languages and propose a MultiPL extension of LLMs utilizing a hybrid mixture of experts (MoE), called MultiPL-MoE. Specifically, MultiPL-MoE combines two paired MoEs to optimize expert selection at both the token and segment levels. The token-level MoE is a standard upcycling MoE structure with a shared expert and a novel gate weight normalization approach that aids in the final fusion with the segment-level MoE. The segment-level MoE incorporates two innovative designs to better capture the syntactic structure and contextual patterns of programming languages: First, using a sliding window to partition the input token sequence into multiple segments; Then, adopting an expert-choice routing strategy that allows experts to select the top-k segments. The results of the experiment proved the effectiveness of MultiPL-MoE.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 19 of its 43 references that are in the map. First posted on arXiv 22 Aug 2025.

Builds on

19 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing21 Aug
  2. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  4. 2024Mixtral of Experts8 Jan
  5. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 11

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  5. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  6. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  7. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  8. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 66
  9. 2023Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translationcited by 59
  10. 2022Residual Mixture of Expertscited by 39
  11. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts
  2. 2026A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMDelta Integration into Upcycled MoE
  3. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  4. 2025Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLM
  5. 2022MultiCoder: Multi-Programming-Lingual Pre-Training for Low-Resource Code Completion
  6. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  7. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
  8. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs