# mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generation

arXiv: https://arxiv.org/abs/2410.15037 · HTML: https://mlsanity.com/paper/2410.15037/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 19 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [Mapping Language to Code in Programmatic Context](https://mlsanity.com/paper/1808.09588/) (2018)
- [Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2309.11925/) (2023)
- [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://mlsanity.com/paper/2105.00572/) (2021)
- [HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization](https://mlsanity.com/paper/2402.16694/) (2024)
- [Measuring The Impact Of Programming Language Distribution](https://mlsanity.com/paper/2302.01973/) (2023)
- [AixBench: A Code Generation Benchmark Dataset](https://mlsanity.com/paper/2206.13179/) (2022)
- [CSEPrompts: A Benchmark of Introductory Computer Science Prompts](https://mlsanity.com/paper/2404.02540/) (2024)

## Built on by (most cited first, 30 of 32)

- [TigerLLM - A Family of Bangla Large Language Models](https://mlsanity.com/paper/2503.10995/) (2025)
- [When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy](https://mlsanity.com/paper/2505.22888/) (2025)
- [Aligning Multimodal LLM with Human Preference: A Survey](https://mlsanity.com/paper/2503.14504/) (2025)
- [TigerCoder: A Novel Suite of LLMs for Code Generation in Bangla](https://mlsanity.com/paper/2509.09101/) (2025)
- [3LM: Bridging Arabic, STEM, and Code through Benchmarking](https://mlsanity.com/paper/2507.15850/) (2025)
- [MojoBench: Language Modeling and Benchmarks for Mojo](https://mlsanity.com/paper/2410.17736/) (2024)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generation](https://mlsanity.com/paper/2508.03678/) (2025)
- [DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows](https://mlsanity.com/paper/2512.04416/) (2025)
- [Code LLMs: A Taxonomy-based Survey](https://mlsanity.com/paper/2412.08291/) (2024)
- [Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Models](https://mlsanity.com/paper/2509.16696/) (2025)
- [PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models](https://mlsanity.com/paper/2510.24792/) (2025)
- [Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation](https://mlsanity.com/paper/2604.04532/) (2026)
- [GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation](https://mlsanity.com/paper/2604.24929/) (2026)
- [ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models](https://mlsanity.com/paper/2510.06014/) (2025)
- [Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices](https://mlsanity.com/paper/2510.24450/) (2025)
- [BanglaSTEM: A Parallel Corpus for Technical Domain Bangla-English Translation](https://mlsanity.com/paper/2511.03498/) (2025)
- [From Prompt to Product: A Human-Centered Benchmark of Agentic App Generation Systems](https://mlsanity.com/paper/2512.18080/) (2025)
- [MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark](https://mlsanity.com/paper/2601.03708/) (2026)
- [Large Language Models for Mental Health: A Multilingual Evaluation](https://mlsanity.com/paper/2602.02440/) (2026)
- [Rethinking Scientific Modeling: Toward Physically Consistent and Simulation-Executable Programmatic Generation](https://mlsanity.com/paper/2602.07083/) (2026)
- [SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation](https://mlsanity.com/paper/2505.24324/) (2025)
- [ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation](https://mlsanity.com/paper/2509.22768/) (2025)
- [Retriv at BLP-2025 Task 2: Test-Driven Feedback-Guided Framework for Bangla-to-Python Code Generation](https://mlsanity.com/paper/2511.07382/) (2025)
- [Enhancing LLM Code Generation Capabilities through Test-Driven Development and Code Interpreter](https://mlsanity.com/paper/2511.12823/) (2025)
- [NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generation](https://mlsanity.com/paper/2511.16787/) (2025)
- [Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation](https://mlsanity.com/paper/2511.17290/) (2025)
- [BanglaForge: LLM Collaboration with Self-Refinement for Bangla Code Generation](https://mlsanity.com/paper/2512.19122/) (2025)
- [PyBangla at BLP-2025 Task 2: Enhancing Bangla-to-Python Code Generation with Iterative Self-Correction and Multilingual Agents](https://mlsanity.com/paper/2512.23713/) (2025)
- [EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems](https://mlsanity.com/paper/2602.10171/) (2026)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) have significantly enhanced code generation from natural language prompts. The HumanEval Benchmark, developed by OpenAI, remains the most widely used code generation benchmark. However, this and other Code LLM benchmarks face critical limitations, particularly in task diversity, test coverage, and linguistic scope. Current evaluations primarily focus on English-to-Python conversion tasks with limited test cases, potentially overestimating model performance. While recent works have addressed test coverage and programming language (PL) diversity, code generation from low-resource language prompts remains largely unexplored. To address this gap, we introduce mHumanEval, an extended benchmark supporting prompts in over 200 natural languages. We employ established machine translation methods to compile the benchmark, coupled with a quality assurance process. Furthermore, we provide expert human translations for 15 diverse natural languages (NLs). We conclude by analyzing the multilingual code generation capabilities of state-of-the-art (SOTA) Code LLMs, offering insights into the current landscape of cross-lingual code generation.