mlsanityData to 14 Oct 2026

MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation

arXiv 2208.08227 ↗First posted 17 Aug 2022cs.LG cs.PLcited by 136

Not on the map. Why?

Abstract (arXiv)

Large language models have demonstrated the ability to generate both natural language and programming language text. Such models open up the possibility of multi-language code generation: could code generation models generalize knowledge from one language to another? Although contemporary code generation models can generate semantically correct Python code, little is known about their abilities with other languages. We propose MultiPL-E, a system for translating unit test-driven code generation benchmarks to new languages. We create the first massively multilingual code generation benchmark by using MultiPL-E to translate two popular Python code generation benchmarks to 18 additional programming languages. We use MultiPL-E to extend the HumanEval benchmark and MBPP benchmark to 18 languages that encompass a range of programming paradigms and popularity. Using these new parallel benchmarks, we evaluate the multi-language performance of three state-of-the-art code generation models: Codex, CodeGen, and InCoder. We find that Codex matches or even exceeds its performance on Python for several other languages. The range of programming languages represented in MultiPL-E allow us to explore the impact of language frequency and language features on model performance. Finally, the MultiPL-E approach of compiling code generation benchmarks to new programming languages is both scalable and extensible, making it straightforward to evaluate new models, benchmarks, and languages.

Key facts. Cited by 136 papers in the map (data through 14 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 17 Aug 2022.

Builds on

31 of 45 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  9. 2018Datasheets for Datasetscited by 1,386
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  11. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  12. 2021Measuring Coding Challenge Competence With APPScited by 902
19 more
  1. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  4. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  5. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  6. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  7. 2022CodeT: Code Generation with Generated Testscited by 384
  8. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  9. 2019SPoC: Search-based Pseudocode to Codecited by 176
  10. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  11. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  12. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  13. 2022Productivity Assessment of Neural Code Completioncited by 98
  14. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  15. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  16. 2020LambdaNet: Probabilistic Type Inference using Graph Neural Networkscited by 44
  17. 2021Multilingual training for Software Engineeringcited by 43
  18. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  19. 2019TypeWriter: Neural Type Prediction with Search-based Validationcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 48; 2025: 39; 2026: 39.

136 papers.

Sort:

By area:

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  6. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  7. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  8. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  9. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  10. 2026MiMo-V2-Flash Technical Reportcited by 191
  11. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
  12. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
38 more
  1. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 123
  2. 2024Knowledge Fusion of Large Language Modelscited by 115
  3. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMscited by 70
  4. 2023ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Codecited by 65
  5. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 64
  6. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  7. 2025LongCat-Flash Technical Reportcited by 58
  8. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  9. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  10. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 47
  11. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulationcited by 45
  12. 2024How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmarkcited by 43
  13. 2024CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repaircited by 42
  14. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  15. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  16. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 37
  17. 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfacescited by 34
  18. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  19. 2024On Leakage of Code Generation Evaluation Datasetscited by 32
  20. 2024Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Modelscited by 30
  21. 2024CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?cited by 30
  22. 2025Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thoughtcited by 29
  23. 2025LongCat-Flash-Omni Technical Reportcited by 26
  24. 2024Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Taskscited by 25
  25. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 25
  26. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 23
  27. 2024CodeEditorBench: Evaluating Code Editing Capability of Large Language Modelscited by 23
  28. 2024Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmarkcited by 23
  29. 2025Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUscited by 22
  30. 2026Olmo Hybrid: From Theory to Practice and Backcited by 22
  31. 2024M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluationcited by 21
  32. 2025Mid-Training of Large Language Models: A Surveycited by 21
  33. 2025A Survey on LLM Mid-Trainingcited by 21
  34. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 21
  35. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 20
  36. 2026Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?cited by 20
  37. 2024UniCoder: Scaling Code Large Language Model via Universal Codecited by 18
  38. 2024On the Limit of Language Models as Planning Formalizerscited by 18

50 most cited of 136 All 136 (JSON)