# CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks

arXiv: https://arxiv.org/abs/2507.05269 · HTML: https://mlsanity.com/paper/2507.05269/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 19 of its 109 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?](https://mlsanity.com/paper/2502.12115/) (2025)
- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG](https://mlsanity.com/paper/2406.11147/) (2024)
- [AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testing](https://mlsanity.com/paper/2409.10737/) (2024)
- [CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning](https://mlsanity.com/paper/2504.14119/) (2025)
- [E&V: Prompting Large Language Models to Perform Static Analysis by Pseudo-code Execution and Verification](https://mlsanity.com/paper/2312.08477/) (2023)
- [Advancing Code Coverage: Incorporating Program Analysis with Large Language Models](https://mlsanity.com/paper/2404.04966/) (2024)
- [LLMExplainer: Large Language Model based Bayesian Inference for Graph Explanation Generation](https://mlsanity.com/paper/2407.15351/) (2024)
- [RFCAudit: An LLM Agent for Functional Bug Detection in Network Protocols](https://mlsanity.com/paper/2506.00714/) (2025)
- [CRQBench: A Benchmark of Code Reasoning Questions](https://mlsanity.com/paper/2408.08453/) (2024)
- [Raw Pointer Rewriting with LLMs for Translating C to Safer Rust](https://mlsanity.com/paper/2505.04852/) (2025)

## Built on by (most cited first, 24 of 24)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level](https://mlsanity.com/paper/2601.03731/) (2026)
- [ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoning](https://mlsanity.com/paper/2603.11226/) (2026)
- [Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Models](https://mlsanity.com/paper/2510.15079/) (2025)
- [Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation](https://mlsanity.com/paper/2512.00215/) (2025)
- [RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models](https://mlsanity.com/paper/2607.25996/) (2026)
- [CTIConnect: A Benchmark for Retrieval-Augmented LLMs over Heterogeneous Cyber Threat Intelligence](https://mlsanity.com/paper/2510.11974/) (2025)
- [Gistify! Codebase-Level Understanding via Runtime Execution](https://mlsanity.com/paper/2510.26790/) (2025)
- [Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing](https://mlsanity.com/paper/2509.14335/) (2025)
- [Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings](https://mlsanity.com/paper/2512.14917/) (2025)
- [Counting Without Running: Evaluating LLMs' Reasoning About Code Complexity](https://mlsanity.com/paper/2512.04355/) (2025)
- [How Robustly do LLMs Understand Execution Semantics?](https://mlsanity.com/paper/2604.16320/) (2026)
- [Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks](https://mlsanity.com/paper/2604.12379/) (2026)
- [PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning](https://mlsanity.com/paper/2605.19382/) (2026)
- [GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory](https://mlsanity.com/paper/2606.03144/) (2026)
- [Detecting Differences Is Not Understanding Structure: Large Language Models Fail at Graph Isomorphism](https://mlsanity.com/paper/2606.09484/) (2026)
- [Grounded AI for Code Review: Resource-Efficient Large-Model Serving in Enterprise Pipelines](https://mlsanity.com/paper/2510.10290/) (2025)
- [CMind: An AI Agent for Localizing C Memory Bugs](https://mlsanity.com/paper/2601.14434/) (2026)
- [The Path Not Taken: Duality in Reasoning about Program Execution](https://mlsanity.com/paper/2604.20917/) (2026)
- [Teaching LLMs Program Semantics via Symbolic Execution Traces](https://mlsanity.com/paper/2605.06184/) (2026)
- [Croissant Tasks: A Metadata Format for Reproducible Machine Learning Evaluations](https://mlsanity.com/paper/2605.29786/) (2026)
- [Codoku: Renewable Program-Reasoning Challenges for Frontier Coding Agents](https://mlsanity.com/paper/2609.34661/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been widely adopted across diverse domains of software engineering, such as code generation, program repair, and vulnerability detection. These applications require understanding beyond surface-level code patterns: value propagation, control flow, and interdependence between program elements. However, existing benchmarks primarily evaluate end-to-end outcomes, such as whether code is correctly repaired or generated, leaving the models' ability for program semantic reasoning underexplored. This work presents CORE, a high-quality, human-verified benchmark designed to evaluate LLMs on fundamental static analysis tasks. CORE includes 12,553 task instances spanning data dependency, control dependency, and information flow across programs written in C/C++, Java, and Python. To ensure semantic diversity and reasoning complexity, we propose a semantics-aware diverse sampling strategy that selects targets and task instances based on structural coverage and dependency depth. We evaluate 10 mainstream LLMs and show that, while they perform well at identifying dependencies, models still struggle with tasks that require deeper semantic understanding and multi-step reasoning. We further conduct qualitative analyses to uncover key challenges, such as complex control structures and backward dependency patterns, offering insights into improving LLMs' code reasoning capabilities.