mlsanityData to 7 Oct 2026

MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages

arXiv 2203.08388 ↗First posted 16 Mar 2022cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

While there has been a recent burgeoning of applications at the intersection of natural and programming languages, such as code generation and code summarization, these applications are usually English-centric. This creates a barrier for program developers who are not proficient in English. To mitigate this gap in technology development across languages, we propose a multilingual dataset, MCoNaLa, to benchmark code generation from natural language commands extending beyond English. Modeled off of the methodology from the English Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896 NL-code pairs in three languages: Spanish, Japanese, and Russian. We present a quantitative evaluation of performance on the MCoNaLa dataset by testing with state-of-the-art code generation systems. While the difficulties vary across these three languages, all systems lag significantly behind their English counterparts, revealing the challenges in adapting code generation to new languages.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 31 of its 62 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

31 of 62 references.

References · 31

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  5. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  6. 2021Measuring Coding Challenge Competence With APPScited by 879
  7. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  9. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  10. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  11. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  12. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
19 more
  1. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  2. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  3. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  4. 2020A Transformer-based Approach for Source Code Summarizationcited by 129
  5. 2018Mapping Language to Code in Programmatic Contextcited by 104
  6. 2017Abstract Syntax Networks for Code Generation and Semantic Parsingcited by 92
  7. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 90
  8. 2018TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generationcited by 88
  9. 2016Latent Predictor Networks for Code Generationcited by 73
  10. 2019Generalized Data Augmentation for Low-Resource Translationcited by 53
  11. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  12. 2020Incorporating External Knowledge through Pre-training for Natural Language to Code Generationcited by 36
  13. 2018StaQC: A Systematically Mined Question-Code Dataset from Stack Overflowcited by 30
  14. 2021Zero-Shot Cross-lingual Semantic Parsingcited by 19
  15. 2020Bootstrapping a Crosslingual Semantic Parsercited by 13
  16. 2019An Empirical Study of C++ Vulnerabilities in Crowd-Sourced Code Examplescited by 7
  17. 2021Code Generation from Natural Language with Less Prior and More Monolingual Datacited by 6
  18. 2021Lyra: A Benchmark for Turducken-Style Code Generationcited by 4
  19. 2021Cross-Lingual Training with Dense Retrieval for Document Retrievalcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 8; 2024: 11; 2025: 14; 2026: 2.

41 papers.

Sort:

By area:

  1. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  2. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  3. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  4. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  5. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  6. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  7. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 80
  8. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  9. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  10. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 46
  11. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40
  12. 2024Automatic Programming: Large Language Models and Beyondcited by 35
29 more
  1. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  2. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  3. 2024AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queriescited by 32
  4. 2022ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languagescited by 29
  5. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
  6. 2024Analyzing the Performance of Large Language Models on Code Summarizationcited by 19
  7. 2025An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunitiescited by 17
  8. 2025How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Codecited by 9
  9. 2023CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generationcited by 8
  10. 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
  11. 2023XSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representationscited by 5
  12. 2024Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codescited by 5
  13. 2025LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languagescited by 5
  14. 2025LLM Code Customization with Visual Results: A Benchmark on TikZcited by 5
  15. 2024Fortran2CPP: Automating Fortran-to-C++ Translation using LLMs via Multi-Turn Dialogue and Dual-Agent Integrationcited by 4
  16. 2025More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generationcited by 4
  17. 2024Training Task Experts through Retrieval Based Distillationcited by 3
  18. 2024Contextualized Data-Wrangling Code Generation in Computational Notebookscited by 3
  19. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  20. 2022Bootstrapping Multilingual Semantic Parsers using Large Language Modelscited by 2
  21. 2023Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsingcited by 2
  22. 2025ThrowBench: Benchmarking LLMs by Predicting Runtime Exceptionscited by 2
  23. 2025StRuCom: A Novel Dataset of Structured Code Comments in Russiancited by 1
  24. 2025CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurementcited by 1
  25. 2026BlueFin: Benchmarking LLM Agents on Financial Spreadsheetscited by 1
  26. 2025ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generationcited by 0
  27. 2025NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generationcited by 0
  28. 2025Scaling Competence, Shrinking Reasoning: Cognitive Signatures in Language Model Learningcited by 0
  29. 2026RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specificationscited by 0