mlsanityData to 7 Oct 2026

Multi-lingual Evaluation of Code Generation Models

arXiv 2210.14868 ↗First posted 26 Oct 2022cs.LG cs.CLcited by 158

Not on the map. Why?

Abstract (arXiv)

We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages and are generated using a scalable conversion framework that transpiles prompts and test cases from the original Python datasets into the corresponding data in the target language. Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages, and zero-shot translation abilities even on mono-lingual settings. Furthermore, we use our code generation model to perform large-scale bootstrapping to obtain synthetic canonical solutions in several languages, which can be used for other code-related evaluations such as code insertion, robustness, or summarization tasks. Overall, our benchmarks represents a significant step towards a deeper understanding of language models' code generation abilities. We publicly release our code and datasets at https://github.com/amazon-research/mxeval.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Builds on 38 of its 73 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

38 of 73 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  12. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
26 more
  1. 2021Measuring Coding Challenge Competence With APPScited by 879
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  3. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  4. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  5. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  6. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  7. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  8. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  9. 2020Unsupervised Translation of Programming Languagescited by 262
  10. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  11. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  12. 2019SPoC: Search-based Pseudocode to Codecited by 176
  13. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  14. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  15. 2022Natural Language to Code Translation with Executioncited by 122
  16. 2018Tree-to-tree Neural Networks for Program Translationcited by 93
  17. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  18. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  19. 2022XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligencecited by 54
  20. 2022CoditT5: Pretraining for Source Code and Natural Language Editingcited by 45
  21. 2022Code Translation with Compiler Representationscited by 42
  22. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  23. 2022CCTEST: Testing and Repairing Code Completion Systemscited by 35
  24. 2019Improving Neural Machine Translation Robustness via Data Augmentation: Beyond Back Translationcited by 16
  25. 2021Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchycited by 14
  26. 2019Introducing MathQA -- A Math-Aware Question Answering Systemcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 36; 2024: 55; 2025: 45; 2026: 18.

158 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2023Teaching Large Language Models to Self-Debugcited by 963
  4. 2024A Survey on Large Language Models for Code Generationcited by 646
  5. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  6. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  7. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  8. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  9. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  10. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  11. 2023SantaCoder: don't reach for the stars!cited by 169
  12. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
38 more
  1. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  2. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  3. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  4. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  5. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  6. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  7. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  8. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  9. 2024SynCode: LLM Generation with Grammar Augmentationcited by 66
  10. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 62
  11. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  12. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 57
  13. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  14. 2023Using Large Language Models to Generate JUnit Tests: An Empirical Studycited by 53
  15. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  16. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 46
  17. 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 46
  18. 2025Seed-Coder: Let the Code Model Curate Data for Itselfcited by 46
  19. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45
  20. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 42
  21. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  22. 2024HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generationcited by 39
  23. 2023SALLM: Security Assessment of Generated Codecited by 35
  24. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Modelscited by 34
  25. 2024SWE-bench-java: A GitHub Issue Resolving Benchmark for Javacited by 34
  26. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  27. 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
  28. 2024AST-T5: Structure-Aware Pretraining for Code Generation and Understandingcited by 32
  29. 2024CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Executioncited by 31
  30. 2024CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?cited by 29
  31. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  32. 2023Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generationcited by 27
  33. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 27
  34. 2024Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distancecited by 26
  35. 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generationcited by 26
  36. 2023AceCoder: Utilizing Existing Code to Enhance Code Generationcited by 25
  37. 2024MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generationcited by 25
  38. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24

50 most cited of 158 All 158 (JSON)