mlsanityData to 14 Oct 2026

Evaluating LLMs Code Reasoning Under Real-World Context

arXiv 2604.12881 ↗First posted 14 Apr 2026cs.SEcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Code reasoning tasks are increasingly crucial to evaluating large language models (LLMs). Yet most existing benchmarks rely on simplistic, LLM-generated snippets or human-written solutions to code challenges and often restrict inputs and outputs to primitive types, failing to reflect the structure and dependencies of real-world projects. These simplifications limit their ability to measure practical generalizability. We present R2Eval1, a benchmark of 135 code reasoning problems drawn from ten widely used Python projects. Unlike prior work, R2Eval serializes compound and custom types, preserving real-world data complexity and enabling a more realistic assessment of LLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 12 of its 24 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

12 of 24 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Models16 Oct
  2. 2025CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning31 May
  3. 2025Co-PatcheR: Collaborative Software Patching with Component(s)-specific Small Reasoning Models25 May
  4. 2024CodeMind: Evaluating Large Language Models for Code Reasoning15 Feb

Other references · 8

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  4. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  5. 2024A Survey on Large Language Models for Code Generationcited by 658
  6. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  7. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  8. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
  2. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  3. 2026Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
  4. 2025Evaluating the Generalization Capabilities of Large Language Models on Code Reasoning
  5. 2025Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?
  6. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
  7. 2025On Code-Induced Reasoning in LLMs
  8. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation