mlsanityData to 14 Oct 2026

CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs

arXiv 2410.01999 ↗First posted 2 Oct 2024cs.SEcited by 18

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Recent advances in Code Large Language Models (CodeLLMs) have primarily focused on open-ended code generation, often overlooking the crucial aspect of code understanding and reasoning. To bridge this gap, we introduce CodeMMLU, a comprehensive multiple-choice benchmark designed to evaluate the depth of software and code comprehension in LLMs. CodeMMLU includes nearly 20,000 questions spanning diverse domains, including code analysis, defect detection, and software engineering principles across multiple programming languages. Unlike traditional benchmarks that emphasize code generation, CodeMMLU assesses a model's ability to reason about programs across a wide-range of tasks such as code repair, execution reasoning, and fill-in-the-blank challenges. Our extensive evaluation reveals that even state-of-the-art models struggle with CodeMMLU, highlighting significant gaps in comprehension beyond generation. By emphasizing the essential connection between code understanding and effective AI-assisted development, CodeMMLU provides a critical resource for advancing more reliable and capable coding assistants.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 58 of its 98 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

58 of 98 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Qwen2.5-Coder Technical Report18 Sep
  2. 2024Stable Code Technical Report1 Apr
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  4. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement22 Feb

Other references · 54

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2024Qwen2 Technical Reportcited by 2,120
  12. 2024Mixtral of Expertscited by 1,823
38 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  6. 2023StarCoder: may the source be with you!cited by 799
  7. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  8. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  9. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  10. 2024Yi: Open Foundation Models by 01.AIcited by 719
  11. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  12. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  13. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  14. 2024InternLM2 Technical Reportcited by 397
  15. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  16. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  17. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  18. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  19. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  20. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  21. 2023SantaCoder: don't reach for the stars!cited by 170
  22. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  23. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 107
  24. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 106
  25. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  26. 2022XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligencecited by 56
  27. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 47
  28. 2024TESTEVAL: Benchmarking Large Language Models for Test Case Generationcited by 40
  29. 2023Automatic Code Summarization via ChatGPT: How Far Are We?cited by 34
  30. 2024On Leakage of Code Generation Evaluation Datasetscited by 32
  31. 2021CodeQA: A Question Answering Dataset for Source Code Comprehensioncited by 30
  32. 2023CodeTF: One-stop Transformer Library for State-of-the-art Code LLMscited by 21
  33. 2023The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generationcited by 20
  34. 2023Better Language Models of Code through Self-Improvementcited by 14
  35. 2019Learning Execution through Neural Code Fusioncited by 13
  36. 2024InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Modelscited by 13
  37. 2022Detect-Localize-Repair: A Unified Framework for Learning to Debug with CodeT5cited by 10
  38. 2022HierarchyNet: Learning to Summarize Source Code with Heterogeneous Representationscited by 9

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 5.

18 papers.

Sort:

By area:

  1. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 59
  2. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  3. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  4. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 11
  5. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
  6. 2025Towards Formal Verification of LLM-Generated Code from Natural Language Promptscited by 8
  7. 2025On Code-Induced Reasoning in LLMscited by 5
  8. 2026RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Modelscited by 5
  9. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  10. 2025Towards Verified Code Reasoning by LLMscited by 4
  11. 2025When Names Disappear: Revealing What LLMs Actually Understand About Codecited by 4
  12. 2025Towards Reliable LLM-based Robot Planning via Combined Uncertainty Estimationcited by 4
6 more
  1. 2026Monitorability as a Free Gift: How RLVR Spontaneously Aligns Reasoningcited by 4
  2. 2024VisualCoder: Guiding Large Language Models in Code Execution with Fine-grained Multimodal Chain-of-Thought Reasoningcited by 3
  3. 2025You Had One Job: Per-Task Quantization Using LLMs' Hidden Representationscited by 3
  4. 2026Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolutioncited by 2
  5. 2026An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoningcited by 1
  6. 2026DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Contextcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  2. 2024CodeMind: Evaluating Large Language Models for Code Reasoning
  3. 2026CodeGlance: Understanding Code Reasoning Challenges in LLMs through Multi-Dimensional Feature Analysis
  4. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
  5. 2025CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning
  6. 2026Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
  7. 2025Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models ↗
  8. 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks