mlsanityData to 7 Oct 2026

McEval: Massively Multilingual Code Evaluation

arXiv 2406.07436 ↗First posted 11 Jun 2024cs.PLcited by 42

Not on the map. Why?

Abstract (arXiv)

Code large language models (LLMs) have shown remarkable advances in code understanding, completion, and generation tasks. Programming benchmarks, comprised of a selection of code challenges and corresponding test cases, serve as a standard to evaluate the capability of different LLMs in such tasks. However, most existing benchmarks primarily focus on Python and are still restricted to a limited number of languages, where other languages are translated from the Python samples (e.g. MultiPL-E) degrading the data diversity. To further facilitate the research of code LLMs, we propose a massively multilingual code benchmark covering 40 programming languages (McEval) with 16K test samples, which substantially pushes the limits of code LLMs in multilingual scenarios. The benchmark contains challenging code completion, understanding, and generation evaluation tasks with finely curated massively multilingual instruction corpora McEval-Instruct. In addition, we introduce an effective multilingual coder mCoder trained on McEval-Instruct to support multilingual programming language generation. Extensive experimental results on McEval show that there is still a difficult journey between open-source models and closed-source LLMs (e.g. GPT-series models) in numerous languages. The instruction corpora, evaluation benchmark, and leaderboard are available at urlhttps://mceval.github.io/.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 21; 2026: 11.

42 papers.

Sort:

By area:

  1. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  2. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  3. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  4. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  5. 2024CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Executioncited by 31
  6. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  7. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  8. 2024Evaluating and Aligning CodeLLMs on Human Preferencecited by 22
  9. 2025OJBench: A Competition Level Code Benchmark For Large Language Modelscited by 22
  10. 2024M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluationcited by 21
  11. 2024ExecRepoBench: Multi-level Executable Code Completion Evaluationcited by 19
  12. 2025CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generationcited by 18
30 more
  1. 2025SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Mannercited by 18
  2. 2024InterTrans: Leveraging Transitive Intermediate Translations to Enhance LLM-based Code Translationcited by 15
  3. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  4. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  5. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  6. 2025MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMscited by 9
  7. 2025Multi-Agent Collaboration for Multilingual Code Instruction Tuningcited by 6
  8. 2026ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Developmentcited by 6
  9. 2025IFEvalCode: Controlled Code Generationcited by 5
  10. 2024Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarkscited by 3
  11. 2025Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generationcited by 3
  12. 2025Generating Automotive Code: Large Language Models for Software Development and Verification in Safety-Critical Systemscited by 3
  13. 2025TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Modelscited by 3
  14. 2025Virtual Width Networkscited by 3
  15. 2025ThrowBench: Benchmarking LLMs by Predicting Runtime Exceptionscited by 2
  16. 2025Is Compression Really Linear with Code Intelligence?cited by 2
  17. 2025Multilingual Multimodal Software Developer for Code Generationcited by 2
  18. 2025Intention Chain-of-Thought Prompting with Dynamic Routing for Code Generationcited by 2
  19. 2026ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocationcited by 2
  20. 2026Multi-LCB: Extending LiveCodeBench to Multiple Programming Languagescited by 2
  21. 2025STEPWISE-CODEX-Bench: Evaluating Complex Multi-Function Comprehension and Fine-Grained Execution Reasoningcited by 1
  22. 2025RPM-MCTS: Knowledge-Retrieval as Process Reward Model with Monte Carlo Tree Search for Code Generationcited by 1
  23. 2026Evaluating and Achieving Controllable Code Completion in Code LLMcited by 1
  24. 2026Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluationcited by 1
  25. 2026Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?cited by 1
  26. 2026Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RLcited by 1
  27. 2026TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answeringcited by 1
  28. 2026Quantize with Confidence? An Empirical Study of Quantization for Code Generationcited by 0
  29. 2026LatentMD: Benchmarking Markdown Boundary Failures in LLM-Generated Textcited by 0
  30. 2026Hunyuan-A13B Technical Reportcited by 0