# McEval: Massively Multilingual Code Evaluation

arXiv: https://arxiv.org/abs/2406.07436 · HTML: https://mlsanity.com/paper/2406.07436/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2024.

## Built on by (most cited first, 30 of 44)

- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models](https://mlsanity.com/paper/2411.04905/) (2024)
- [Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent](https://mlsanity.com/paper/2411.02937/) (2024)
- [A Comparative Study on Reasoning Patterns of OpenAI's o1 Model](https://mlsanity.com/paper/2410.13639/) (2024)
- [CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2408.13001/) (2024)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [OJBench: A Competition Level Code Benchmark For Large Language Models](https://mlsanity.com/paper/2506.16395/) (2025)
- [Evaluating and Aligning CodeLLMs on Human Preference](https://mlsanity.com/paper/2412.05210/) (2024)
- [M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation](https://mlsanity.com/paper/2410.21157/) (2024)
- [ExecRepoBench: Multi-level Executable Code Completion Evaluation](https://mlsanity.com/paper/2412.11990/) (2024)
- [SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner](https://mlsanity.com/paper/2506.09003/) (2025)
- [CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation](https://mlsanity.com/paper/2502.19166/) (2025)
- [InterTrans: Leveraging Transitive Intermediate Translations to Enhance LLM-based Code Translation](https://mlsanity.com/paper/2411.01063/) (2024)
- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents](https://mlsanity.com/paper/2505.05283/) (2025)
- [MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs](https://mlsanity.com/paper/2511.07250/) (2025)
- [Multi-Agent Collaboration for Multilingual Code Instruction Tuning](https://mlsanity.com/paper/2502.07487/) (2025)
- [ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development](https://mlsanity.com/paper/2601.11077/) (2026)
- [IFEvalCode: Controlled Code Generation](https://mlsanity.com/paper/2507.22462/) (2025)
- [Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks](https://mlsanity.com/paper/2410.14766/) (2024)
- [Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generation](https://mlsanity.com/paper/2506.03535/) (2025)
- [Generating Automotive Code: Large Language Models for Software Development and Verification in Safety-Critical Systems](https://mlsanity.com/paper/2506.04038/) (2025)
- [TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models](https://mlsanity.com/paper/2506.18421/) (2025)
- [Multilingual Multimodal Software Developer for Code Generation](https://mlsanity.com/paper/2507.08719/) (2025)
- [Virtual Width Networks](https://mlsanity.com/paper/2511.11238/) (2025)
- [ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocation](https://mlsanity.com/paper/2601.21420/) (2026)
- [ThrowBench: Benchmarking LLMs by Predicting Runtime Exceptions](https://mlsanity.com/paper/2503.04241/) (2025)
- [Is Compression Really Linear with Code Intelligence?](https://mlsanity.com/paper/2505.11441/) (2025)

## Abstract (arXiv)

Code large language models (LLMs) have shown remarkable advances in code understanding, completion, and generation tasks. Programming benchmarks, comprised of a selection of code challenges and corresponding test cases, serve as a standard to evaluate the capability of different LLMs in such tasks. However, most existing benchmarks primarily focus on Python and are still restricted to a limited number of languages, where other languages are translated from the Python samples (e.g. MultiPL-E) degrading the data diversity. To further facilitate the research of code LLMs, we propose a massively multilingual code benchmark covering 40 programming languages (McEval) with 16K test samples, which substantially pushes the limits of code LLMs in multilingual scenarios. The benchmark contains challenging code completion, understanding, and generation evaluation tasks with finely curated massively multilingual instruction corpora McEval-Instruct. In addition, we introduce an effective multilingual coder mCoder trained on McEval-Instruct to support multilingual programming language generation. Extensive experimental results on McEval show that there is still a difficult journey between open-source models and closed-source LLMs (e.g. GPT-series models) in numerous languages. The instruction corpora, evaluation benchmark, and leaderboard are available at urlhttps://mceval.github.io/.