mlsanityData to 14 Oct 2026

ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages

arXiv 2212.06742 ↗First posted 13 Dec 2022cs.CL cs.LG cs.PLcited by 29

Not on the map. Why?

Abstract (arXiv)

Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We release our code and pre-trained checkpoints.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 35 of its 53 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

35 of 53 references.

References · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
23 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  4. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  6. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  7. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  8. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  10. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  11. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  13. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  14. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  15. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  16. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  17. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
  18. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  19. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 108
  20. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  21. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 75
  22. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  23. 2020Highway Transformer: Self-Gating Enhanced Self-Attentive Networkscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 16; 2025: 4; 2026: 2.

29 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  3. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  4. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 86
  5. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  6. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  7. 2023Zero-Shot Detection of Machine-Generated Codescited by 24
  8. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 23
  9. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  10. 2023Tool-Augmented Reward Modelingcited by 20
  11. 2024Tokenization Falling Short: On Subword Robustness in Large Language Modelscited by 20
  12. 2024UniCoder: Scaling Code Large Language Model via Universal Codecited by 18
17 more
  1. 2024Code Needs Comments: Enhancing Code LLMs with Comment Augmentationcited by 16
  2. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actionscited by 15
  3. 2025Curiosity-Driven Reinforcement Learning from Human Feedbackcited by 15
  4. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  5. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  6. 2024Autoregressive Pre-Training on Pixels and Textscited by 10
  7. 2024FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Datacited by 10
  8. 2025CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languagescited by 10
  9. 2024On Training Data Influence of GPT Modelscited by 9
  10. 2024RedStone: Curating General, Code, Math, and QA Data for Large Language Modelscited by 6
  11. 2024REDO: Execution-Free Runtime Error Detection for COding Agentscited by 2
  12. 2025Debiasing Multilingual LLMs in Cross-lingual Latent Spacecited by 2
  13. 2024Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfercited by 1
  14. 2023Safurai 001: New Qualitative Approach for Code LLM Evaluationcited by 0
  15. 2023Safurai-Csharp: Harnessing Synthetic Data to improve language-specific Code LLMcited by 0
  16. 2026Prompt-Driven Code Summarization: A Systematic Literature Reviewcited by 0
  17. 2026Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Qualitycited by 0