# From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level

arXiv: https://arxiv.org/abs/2601.03731 · HTML: https://mlsanity.com/paper/2601.03731/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 10 of 10)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [Reasoning Runtime Behavior of a Program with LLM: How Far Are We?](https://mlsanity.com/paper/2403.16437/) (2024)
- [Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation](https://mlsanity.com/paper/2406.14644/) (2024)
- [CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks](https://mlsanity.com/paper/2507.05269/) (2025)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)

## Built on by (most cited first, 6 of 6)

- [CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions](https://mlsanity.com/paper/2602.20213/) (2026)
- [Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA](https://mlsanity.com/paper/2605.29277/) (2026)
- [RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists](https://mlsanity.com/paper/2608.04783/) (2026)
- [Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair](https://mlsanity.com/paper/2605.07276/) (2026)
- [Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt-Engineering Quality Assurance](https://mlsanity.com/paper/2605.12280/) (2026)
- [ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning](https://mlsanity.com/paper/2607.25369/) (2026)

## Abstract (arXiv)

As large language models (LLMs) evolve into autonomous agents, evaluating repository-level reasoning, the ability to maintain logical consistency across massive, real-world, interdependent file systems, has become critical. Current benchmarks typically fluctuate between isolated code snippets and black-box evaluations. We present RepoReason, a white-box diagnostic benchmark centered on abductive assertion verification. To eliminate memorization while preserving authentic logical depth, we implement an execution-driven mutation framework that utilizes the environment as a semantic oracle to regenerate ground-truth states. Furthermore, we establish a fine-grained diagnostic system using dynamic program slicing, quantifying reasoning via three orthogonal metrics: ESV (reading load), MCL (simulation depth), and DFI (integration width). Comprehensive evaluations of frontier models (e.g., Claude-4.5-Sonnet, DeepSeek-v3.1-Terminus) reveal a prevalent aggregation deficit, where integration width serves as the primary cognitive bottleneck. Our findings provide granular white-box insights for optimizing the next generation of agentic software engineering.