mlsanityData to 14 Oct 2026

Mix-of-Language-Experts Architecture for Multilingual Programming

arXiv 2506.18923 ↗First posted 18 Jun 2025cs.PL cs.CL cs.SEcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in aiding developers with tasks like code comprehension, generation, and translation. Supporting multilingual programming -- i.e., coding tasks across multiple programming languages -- typically requires either (1) finetuning a single LLM across all programming languages, which is cost-efficient but sacrifices language-specific specialization and performance, or (2) finetuning separate LLMs for each programming language, which allows for specialization but is computationally expensive and storage-intensive due to the duplication of parameters. This paper introduces MoLE (Mix-of-Language-Experts), a novel architecture that balances efficiency and specialization for multilingual programming. MoLE is composed of a base model, a shared LoRA (low-rank adaptation) module, and a collection of language-specific LoRA modules. These modules are jointly optimized during the finetuning process, enabling effective knowledge sharing and specialization across programming languages. During inference, MoLE automatically routes to the language-specific LoRA module corresponding to the programming language of the code token being generated. Our experiments demonstrate that MoLE achieves greater parameter efficiency compared to training separate language-specific LoRAs, while outperforming a single shared LLM finetuned for all programming languages in terms of accuracy.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

36 of 47 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2024Mixtral of Expertscited by 1,823
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  9. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  12. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
24 more
  1. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  2. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  4. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  5. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  6. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  7. 2020Unsupervised Translation of Programming Languagescited by 263
  8. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  9. 2023From Sparse to Soft Mixtures of Expertscited by 248
  10. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  11. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  12. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  13. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  14. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  15. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  16. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 76
  17. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  18. 2024Upcycling Large Language Models into Mixture of Expertscited by 49
  19. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  20. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19
  21. 2023One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarizationcited by 16
  22. 2024InterTrans: Leveraging Transitive Intermediate Translations to Enhance LLM-based Code Translationcited by 16
  23. 2023Multilingual Code Co-Evolution Using Large Language Modelscited by 9
  24. 2024MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Grow Up and Merge: Scaling Strategies for Efficient Language Adaptationcited by 3
  2. 2026COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Samplingcited by 0
  3. 2026LangSelect: Cost-Aware Target-Language Routing for LLM Code Generationcited by 0