mlsanityData to 7 Oct 2026

MultiCoder: Multi-Programming-Lingual Pre-Training for Low-Resource Code Completion

arXiv 2212.09666 ↗First posted 19 Dec 2022cs.CLcited by 6

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Code completion is a valuable topic in both academia and industry. Recently, large-scale mono-programming-lingual (MonoPL) pre-training models have been proposed to boost the performance of code completion. However, the code completion on low-resource programming languages (PL) is difficult for the data-driven paradigm, while there are plenty of developers using low-resource PLs. On the other hand, there are few studies exploring the effects of multi-programming-lingual (MultiPL) pre-training for the code completion, especially the impact on low-resource programming languages. To this end, we propose the MultiCoder to enhance the low-resource code completion via MultiPL pre-training and MultiPL Mixture-of-Experts (MoE) layers. We further propose a novel PL-level MoE routing strategy (PL-MoE) for improving the code completion on all PLs. Experimental results on CodeXGLUE and MultiCC demonstrate that 1) the proposed MultiCoder significantly outperforms the MonoPL baselines on low-resource programming languages, and 2) the PL-MoE module further boosts the performance on six programming languages. In addition, we analyze the effects of the proposed method in details and explore the effectiveness of our method in a variety of scenarios.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 29 of its 39 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

29 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  11. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  12. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
15 more
  1. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  2. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  3. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  4. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  5. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  6. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  7. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  8. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  9. 2019Multilingual is not enough: BERT for Finnishcited by 118
  10. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  11. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  12. 2020Multi-task Learning based Pre-trained Language Model for Code Completioncited by 58
  13. 2022CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequencescited by 49
  14. 2021PyTorrent: A Python Library Corpus for Large-scale Language Modelscited by 8
  15. 2019Zero-Shot Paraphrase Generation with Multilingual Language Modelscited by 7

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 2.

6 papers.

  1. 2024Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chatcited by 23
  2. 2023The potential of LLMs for coding with low-resource and domain-specific programming languagescited by 11
  3. 2025SLMFix: Leveraging Small Language Models for Domain Specific Language Error Fixing with Reinforcement Learningcited by 2
  4. 2024Automatic Detection of LLM-Generated Code: A Comparative Case Study of Contemporary Models Across Function and Class Granularitiescited by 1
  5. 2025NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptationcited by 1
  6. 2024Investigating the Transferability of Code Repair for Low-Resource Programming Languagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts
  2. 2025Seed-Coder: Let the Code Model Curate Data for Itself
  3. 2026CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
  4. 2023CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models
  5. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring
  6. 2024Prompt-based Code Completion via Multi-Retrieval Augmented Generation
  7. 2025Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLM
  8. 2023Multi-Method Self-Training: Improving Code Generation With Text, And Vice Versa