mlsanityData to 7 Oct 2026

CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

arXiv 2401.03065 ↗First posted 5 Jan 2024cs.SE cs.AI cs.LGcited by 294

Not on the map. Why?

Cited by 95 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present CRUXEval (Code Reasoning, Understanding, and eXecution Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each function comes with an input-output pair, leading to two natural tasks: input prediction and output prediction. First, we propose a generic recipe for generating our execution benchmark which can be used to create future variation of the benchmark. Second, we evaluate twenty code models on our benchmark and discover that many recent high-scoring models on HumanEval do not show the same improvements on our benchmark. Third, we show that simple CoT and fine-tuning schemes can improve performance on our benchmark but remain far from solving it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75% and 81% on input and output prediction, respectively. In contrast, Code Llama 34B achieves a pass@1 of 50% and 46% on input and output prediction, highlighting the gap between open and closed source models. As no model is close to acing CRUXEval, we provide examples of consistent GPT-4 failures on simple programs as a lens into its code reasoning capabilities and areas for improvement.

Key facts. Cited by 294 papers in the map (data through 7 Oct 2026). Builds on 88 of its 109 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

88 of 109 references.

References · 88

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2023Mistral 7Bcited by 3,065
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
38 more
  1. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  3. 2023Teaching Large Language Models to Self-Debugcited by 963
  4. 2021Measuring Coding Challenge Competence With APPScited by 879
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  7. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  12. 2023Textbooks Are All You Needcited by 561
  13. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  14. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  15. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  16. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  17. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  18. 2022CodeT: Code Generation with Generated Testscited by 369
  19. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  20. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  21. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  22. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  23. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  24. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  25. 2020Unsupervised Translation of Programming Languagescited by 262
  26. 2023The False Promise of Imitating Proprietary LLMscited by 254
  27. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  28. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  29. 2023Looped Transformers as Programmable Computerscited by 224
  30. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  31. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  32. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  33. 2023Planning with Large Language Models for Code Generationcited by 179
  34. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  35. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 177
  36. 2023SantaCoder: don't reach for the stars!cited by 169
  37. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  38. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 42; 2025: 131; 2026: 121.

294 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  4. 2025Kimi K2: Open Agentic Intelligencecited by 523
  5. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  6. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  7. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  8. 2025Learning to Reason without External Rewardscited by 199
  9. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  10. 2026MiMo-V2-Flash Technical Reportcited by 163
  11. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  12. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
38 more
  1. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  2. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  3. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  4. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  5. 2025S*: Test Time Scaling for Code Generationcited by 95
  6. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 87
  7. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  8. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  9. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 75
  10. 2024Self-Taught Evaluatorscited by 70
  11. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 64
  12. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  13. 2026Qwen3-Coder-Next Technical Reportcited by 60
  14. 2025LongCat-Flash Technical Reportcited by 54
  15. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  16. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  17. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
  18. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 48
  19. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 48
  20. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  21. 2025Seed-Coder: Let the Code Model Curate Data for Itselfcited by 46
  22. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44
  23. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 44
  24. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generationcited by 41
  25. 2026Code as Agent Harnesscited by 40
  26. 2024HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generationcited by 39
  27. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 39
  28. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  29. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  30. 2025Optimizing Length Compression in Large Reasoning Modelscited by 37
  31. 2025dInfer: An Efficient Inference Framework for Diffusion Language Modelscited by 33
  32. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  33. 2024CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Executioncited by 31
  34. 2024CodeMind: Evaluating Large Language Models for Code Reasoningcited by 30
  35. 2024Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Modelscited by 29
  36. 2024CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?cited by 29
  37. 2024Multiple-Choice Questions are Efficient and Robust LLM Evaluatorscited by 28
  38. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28

50 most cited of 294 All 294 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  2. 2025L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution
  3. 2025GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities
  4. 2024CRQBench: A Benchmark of Code Reasoning Questions
  5. 2026ISA-Bench: A Benchmark for Computational Reasoning Across Instruction Set Architectures
  6. 2026PyINE: A Framework for Scalable Elicitation and Oversight via Code Execution
  7. 2025CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction
  8. 2025Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python