mlsanityData to 7 Oct 2026

Reasoning Runtime Behavior of a Program with LLM: How Far Are We?

arXiv 2403.16437 ↗First posted 25 Mar 2024cs.SE cs.CLcited by 44

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities. To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval). Code reasoning is one of the most essential abilities of code LLMs, but existing benchmarks for code reasoning are not sufficient. Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior during program execution, as well as the logical consistency (e.g., the model should not give the correct output if the prediction of execution path is wrong) when performing the reasoning. To address these problems, in this paper, we propose a framework, namely REval, for evaluating code reasoning abilities and consistency of code LLMs with program execution. We utilize existing code benchmarks and adapt them to new benchmarks within our framework. A large-scale empirical study is conducted and most LLMs show unsatisfactory performance on both Runtime Behavior Reasoning (i.e., an average accuracy of 44.4%) and Incremental Consistency Evaluation (i.e., an average IC score of 10.3). Evaluation results of current code LLMs reflect the urgent need for the community to strengthen the code reasoning capability of code LLMs. Our code, data, and newname leaderboard are available at https://r-eval.github.io.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 45 of its 66 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

45 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Teaching Large Language Models to Self-Debug11 Apr
  2. 2023LEVER: Learning to Verify Language-to-Code Generation with Execution16 Feb

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2023Mistral 7Bcited by 3,065
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  12. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
31 more
  1. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  2. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  4. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  5. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  6. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  7. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  8. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  9. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  10. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  11. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  12. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  13. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  14. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  15. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  16. 2023Learning Performance-Improving Code Editscited by 103
  17. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  18. 2023Large Language Models for Test-Free Fault Localizationcited by 67
  19. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  20. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 46
  21. 2023Consistency Analysis of ChatGPTcited by 37
  22. 2023Code Execution with Pre-trained Language Modelscited by 33
  23. 2023TRACED: Execution-aware Pre-training for Source Codecited by 27
  24. 2024Unsupervised Evaluation of Code LLMs with Round-Trip Correctnesscited by 23
  25. 2022Unpacking Large Language Models with Conceptual Consistencycited by 18
  26. 2024Code Simulation Challenges for Large Language Modelscited by 15
  27. 2021Static Inference Meets Deep Learning: A Hybrid Type Inference Approach for Pythoncited by 12
  28. 2023Predicting Code Coverage without Executioncited by 9
  29. 2024DevEval: Evaluating Code Generation in Practical Software Projectscited by 9
  30. 2018Code Vectors: Understanding Programs Through Embedded Abstracted Symbolic Tracescited by 7
  31. 2023LExecutor: Learning-Guided Executioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 25; 2026: 16.

44 papers.

Sort:

By area:

  1. 2024CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Executioncited by 31
  2. 2025Revisit Self-Debugging with Self-Generated Tests for Code Generationcited by 21
  3. 2024NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variationscited by 19
  4. 2025Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inferencecited by 14
  5. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoningcited by 14
  6. 2025Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Callingcited by 13
  7. 2025CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoningcited by 12
  8. 2026Can We Predict Before Executing Machine Learning Agents?cited by 10
  9. 2024TransAgent: Enhancing LLM-Based Code Translation via Fine-Grained Execution Alignmentcited by 7
  10. 2025CoCoNUT: Structural Code Understanding does not fall out of a treecited by 7
  11. 2025HoarePrompt: Structural Reasoning About Program Correctness in Natural Languagecited by 7
  12. 2026From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Levelcited by 6
32 more
  1. 2025L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Executioncited by 5
  2. 2025Large Language Models for Validating Network Protocol Parserscited by 5
  3. 2025LLMs Lean on Priors, Not Programming Language Semanticscited by 5
  4. 2026ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoningcited by 5
  5. 2026RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Modelscited by 5
  6. 2025The Hitchhiker's Guide to Program Analysis, Part II: Deep Thoughts by LLMscited by 4
  7. 2025CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learningcited by 4
  8. 2025When Names Disappear: Revealing What LLMs Actually Understand About Codecited by 4
  9. 2025Gistify! Codebase-Level Understanding via Runtime Executioncited by 4
  10. 2026Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibilitycited by 4
  11. 2025Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditingcited by 3
  12. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settingscited by 3
  13. 2025Is Compression Really Linear with Code Intelligence?cited by 2
  14. 2025Evaluating Software Process Models for Multi-Agent Class-Level Code Generationcited by 2
  15. 2025Investigating Execution-Aware Language Models for Code Optimizationcited by 1
  16. 2025A Semantic-based Optimization Approach for Repairing LLMs: Case Study on Code Generationcited by 1
  17. 2025Re-Evaluating Code LLM Benchmarks Under Semantic Mutationcited by 1
  18. 2025Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMscited by 1
  19. 2025Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agentcited by 1
  20. 2026An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoningcited by 1
  21. 2026StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learningcited by 1
  22. 2026Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binariescited by 1
  23. 2026Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learningcited by 1
  24. 2025Will AI replace Software Engineers? Do not hold your breathcited by 0
  25. 2025Themisto: Jupyter-Based Runtime Benchmarkcited by 0
  26. 2026Coding in a Bubble? Evaluating LLMs in Resolving Context Adaptation Bugs During Code Adaptationcited by 0
  27. 2026ExplainBench: Evaluating Code Explanations from Agentscited by 0
  28. 2026SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvementscited by 0
  29. 2026AgentExecutor: Partial Code Execution via Agentic Context Generationcited by 0
  30. 2026Predicting Program Exit Code with LLMs and Programming Language Semanticscited by 0
  31. 2026Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmarkcited by 0
  32. 2026Codoku: Renewable Program-Reasoning Challenges for Frontier Coding Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
  2. 2025A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models
  3. 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks
  4. 2026CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction
  5. 2026How Robustly do LLMs Understand Execution Semantics?
  6. 2024CodeMind: Evaluating Large Language Models for Code Reasoning
  7. 2025Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models
  8. 2026Evaluating LLMs Code Reasoning Under Real-World Context