Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action
On the map: LLM reasoning → Reasoning for agents, planning and code
Abstract (arXiv)
The rise of large language models (LLMs) has led to dramatic improvements across a wide range of natural language tasks. Their performance on certain tasks can be further enhanced by incorporating test-time reasoning techniques. These inference-time advances have been adopted into the code domain, enabling complex software engineering (SWE) tasks such as code generation, test generation and issue resolution. However, the impact of different reasoning techniques on code-centric SWE tasks has not been systematically explored. In this work, we survey code reasoning techniques that underpin these capabilities, with a focus on test-time compute and inference-time reasoning paradigms. We examine a variety of code-specific reasoning methods and progressively build up to SWE agents, which combine planning, tool use, and multi-step interaction. We also compare the impact of different techniques on coding tasks, highlighting their relative importance and outlining open challenges and future research directions. Across commonly used models and benchmarks, we find that approaches exploiting code-specific signals (e.g., structure and execution feedback) are frequently associated with improved performance, motivating a dedicated study of code reasoning beyond natural-language reasoning.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 82 of its 126 references that are in the map. First posted on arXiv 16 Jun 2025.
Builds on
82 of 126 references.
Closest ancestors · 15
Same direction, newest first
- 2025TRAIL: Trace Reasoning and Agentic Issue Localization13 May
- 2025MSCoT: Structured Chain-of-Thought Generation for Multiple Programming Languages14 Apr
- 2025MoT: Modularization-of-Thought Prompting for Effective Code Generation16 Mar
- 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs26 Feb
- 2025S*: Test Time Scaling for Code Generation20 Feb
- 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesis3 Feb
- 2025A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models30 Jan
- 2024OpenAI o1 System Card21 Dec
- 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation19 Dec
- 2024CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models7 Nov
- 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs2 Oct
- 2024Qwen2.5-Coder Technical Report18 Sep
3 more
Other references · 67
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
- 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
- 2021Measuring Coding Challenge Competence With APPScited by 902
38 more
- 2022A Survey on In-context Learningcited by 721
- 2024A Survey on Large Language Models for Code Generationcited by 658
- 2022Towards Reasoning in Large Language Models: A Surveycited by 596
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
- 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
- 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
- 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
- 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
- 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
- 2022Reasoning with Language Model Prompting: A Surveycited by 252
- 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
- 2023Large Language Model Guided Tree-of-Thoughtcited by 221
- 2025Survey on Evaluation of LLM-based Agentscited by 204
- 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
- 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
- 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
- 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
- 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 152
- 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
- 2022Large Language Models Meet NL2Code: A Surveycited by 138
- 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 133
- 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinementcited by 132
- 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 103
- 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 91
- 2025SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolutioncited by 90
- 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 86
- 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 85
- 2024Planning In Natural Language Improves LLM Search For Code Generationcited by 83
- 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 83
- 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
- 2024Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvementcited by 61
- 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
- 2024MASAI: Modular Architecture for Software-engineering AI Agentscited by 49
- 2024AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodologycited by 46
- 2024A Survey on Evaluating Large Language Models in Code Generation Taskscited by 44
- 2024LeDex: Training LLMs to Better Self-Debug and Explain Codecited by 40
- 2023ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarificationcited by 38
- 2023Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generationcited by 27
50 most cited of 67 All 67 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 2.
4 papers.
- 2026GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPOcited by 6
- 2025An Empirical Study of Reasoning Steps in Thinking Code LLMscited by 2
- 2026How Robustly do LLMs Understand Execution Semantics?cited by 2
- 2025Transformers Can Learn Connectivity in Some Graphs but Not Otherscited by 0
- 2026How Robustly do LLMs Understand Execution Semantics?24 Feb 2026
- 2026GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPO11 Jan 2026
- 2025An Empirical Study of Reasoning Steps in Thinking Code LLMs8 Nov 2025
- 2025Transformers Can Learn Connectivity in Some Graphs but Not Others26 Sep 2025
Similar, not cited
Close in wording, no citation either way
- 2025CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning
- 2025Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents
- 2026TDD-Agent: Test-Driven Reasoning for Code Generation
- 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
- 2025textttReMind: Understanding Deductive Code Reasoning in LLMs
- 2026Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skills
- 2025Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
- 2026FastCode: Fast and Cost-Efficient Code Understanding and Reasoning