mlsanityData to 7 Oct 2026

HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization

arXiv 2402.16694 ↗First posted 26 Feb 2024cs.CL cs.PL cs.SEcited by 84

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to address this deficiency. HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs), and comprises of a collection of 22,080 prompts with an average of 8.33 test cases. By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs. Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation. We make our evaluation code and data publicly available at urlhttps://github.com/FloatAI/humaneval-xl.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 16 of its 21 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

16 of 21 references.

References · 16

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2021Measuring Coding Challenge Competence With APPScited by 879
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2023StarCoder: may the source be with you!cited by 790
  9. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  11. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  12. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
4 more
  1. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  2. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  3. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  4. 2022ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languagescited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 41; 2026: 23.

84 papers.

Sort:

By area:

  1. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 44
  2. 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
  3. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  4. 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generationcited by 26
  5. 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contaminationcited by 22
  6. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 21
  7. 2024ExecRepoBench: Multi-level Executable Code Completion Evaluationcited by 19
  8. 2024Tokenization Falling Short: On Subword Robustness in Large Language Modelscited by 18
  9. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devicescited by 17
  10. 2025FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluationcited by 17
  11. 2024MdEval: Massively Multilingual Code Debuggingcited by 16
  12. 2025DependEval: Benchmarking LLMs for Repository Dependency Understandingcited by 16
38 more
  1. 2025A benchmark for vericoding: formally verified program synthesiscited by 16
  2. 2024InterTrans: Leveraging Transitive Intermediate Translations to Enhance LLM-based Code Translationcited by 15
  3. 2024Concept Space Alignment in Multilingual LLMscited by 13
  4. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  5. 2026Beyond Static Tools: Test-Time Tool Evolution for Scientific Reasoningcited by 12
  6. 2024LLMs: A Game-Changer for Software Engineers?cited by 11
  7. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  8. 2025Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluationcited by 9
  9. 2024HLB: Benchmarking LLMs' Humanlikeness in Language Usecited by 8
  10. 2025CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languagescited by 8
  11. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  12. 2025Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capabilitycited by 7
  13. 2025The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilitiescited by 7
  14. 2025Large Language Model for Verilog Code Generation: Literature Review and the Road Aheadcited by 7
  15. 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
  16. 2024Synthetic Programming Elicitation for Text-to-Code in Very Low-Resource Programming and Formal Languagescited by 6
  17. 2025OSS-Bench: Benchmark Generator for Coding LLMscited by 6
  18. 20253LM: Bridging Arabic, STEM, and Code through Benchmarkingcited by 5
  19. 2025IFEvalCode: Controlled Code Generationcited by 5
  20. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  21. 2024MojoBench: Language Modeling and Benchmarks for Mojocited by 4
  22. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  23. 2025More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generationcited by 4
  24. 2025AppForge: From Assistant to Independent Developer -- Are GPTs Ready for Software Development?cited by 4
  25. 2024Multi-Programming Language Ensemble for Code Generation in Large Language Modelcited by 3
  26. 2024An evaluation of LLM code generation capabilities through graded exercisescited by 3
  27. 2024Code LLMs: A Taxonomy-based Surveycited by 3
  28. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  29. 2025PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Modelscited by 3
  30. 2025PerfCoder: Large Language Models for Interpretable Code Performance Optimizationcited by 3
  31. 2026AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generationcited by 3
  32. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  33. 2026A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inferencecited by 3
  34. 2024Diagnosing and Remedying Knowledge Deficiencies in LLMs via Label-free Curricular Meaningful Learningcited by 2
  35. 2025Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratchcited by 2
  36. 2025MULTITAT: Benchmarking Multilingual Table-and-Text Question Answeringcited by 2
  37. 2025Is Compression Really Linear with Code Intelligence?cited by 2
  38. 2025Evaluating LLMs on Sequential API Call Through Automated Test Generationcited by 2

50 most cited of 84 All 84 (JSON)