mlsanityData to 14 Oct 2026

CRQBench: A Benchmark of Code Reasoning Questions

arXiv 2408.08453 ↗First posted 15 Aug 2024cs.SEcited by 6

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large Language Models have demonstrated exceptional proficiency on coding tasks, but it is challenging to precisely evaluate their code reasoning ability. Existing benchmarks are insufficient as they are unrealistic and conflate semantic reasoning ability with performance on software engineering tasks. We introduce CRQBench, a benchmark of 100 C++ code reasoning questions and answers derived from contextualized code review comments. To curate CRQBench, we use an LLM assistant alongside human inspection, reducing manual effort. We conduct an evaluation of GPT-4 on CRQBench and find that it produces correct responses grounded in the given context for 65 of the 100 questions.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 11 of its 12 references that are in the map. First posted on arXiv 15 Aug 2024.

Builds on

11 of 12 references.

References · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2023The Falcon Series of Open Language Modelscited by 535
  10. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  11. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 73

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Taskscited by 24
  2. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 4
  3. 2025Towards Verified Code Reasoning by LLMscited by 4
  4. 2025QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecturecited by 3
  5. 2026Evaluating LLM Agents on Automated Software Analysis Taskscited by 0
  6. 2026Complexity-Aware Evaluation of LLM Comprehensioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique
  2. 2025CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models
  3. 2026Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
  4. 2025Think Like Human Developers: Harnessing Community Knowledge for Structured Code Reasoning
  5. 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
  6. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
  7. 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs
  8. 2026SWE-QA: A Dataset and Benchmark for Complex Code Understanding