mlsanityData to 7 Oct 2026

A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models

arXiv 2501.18482 ↗First posted 30 Jan 2025cs.SEcited by 6

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Code Executing Reasoning is becoming a new non-functional metric that assesses the ability of large language models (LLMs) in programming tasks. State-of-the-art frameworks (CodeMind or REval) and benchmarks (CruxEval) usually focus on LLM's prediction of a given code's input/output or intermediate variable states/values on limited programs. However, there is no tool for more in-depth analysis of the results. Without such a tool, the observations about LLM's code execution reasoning cannot be generalized to more datasets, preventing the research community and practitioners from devising the next generation of LLMs with better code execution reasoning abilities. This paper introduces ExeRScope, a series of tools and heuristics to analyze the result of code execution reasoning frameworks to understand better the impact of code properties in the studied benchmarks on the code execution reasoning. With such tooling, analysis can be generalized to code with similar properties without the urgent need to design more benchmarks, which is a cumbersome effort.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 28 of its 39 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

28 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning3 Jun
  2. 2024NExT: Teaching Large Language Models to Reason about Code Execution23 Apr
  3. 2023Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain21 Oct
  4. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change21 Jun

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  10. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  11. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  12. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
12 more
  1. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  2. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  3. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 155
  4. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  5. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  6. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  7. 2024Can LLM Graph Reasoning Generalize beyond Pattern Memorization?cited by 46
  8. 2023Code Execution with Pre-trained Language Modelscited by 33
  9. 2023The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Pythoncited by 29
  10. 2024What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?cited by 22
  11. 2024Code Simulation Challenges for Large Language Modelscited by 15
  12. 2024Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursioncited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2026From Assistant to Double Agent: Formalizing and Benchmarking Attacks on OpenClaw for Personalized Local AI Agentcited by 27
  2. 2025Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Modelscited by 5
  3. 2025Assessing the Impact of Code Changes on the Fault Localizability of Large Language Modelscited by 4
  4. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  5. 2026Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibilitycited by 4
  6. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settingscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024CodeMind: Evaluating Large Language Models for Code Reasoning
  2. 2026Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
  3. 2025Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
  4. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?
  5. 2026CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction
  6. 2025Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models
  7. 2026An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoning
  8. 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks