# Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute

arXiv: https://arxiv.org/abs/2503.23803 · HTML: https://mlsanity.com/paper/2503.23803/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 29 of its 53 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 29 of 29)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution](https://mlsanity.com/paper/2502.18449/) (2025)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution](https://mlsanity.com/paper/2501.05040/) (2025)
- [At Which Training Stage Does Code Data Help LLMs Reasoning?](https://mlsanity.com/paper/2309.16298/) (2023)
- [RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation](https://mlsanity.com/paper/2402.16667/) (2024)
- [Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement](https://mlsanity.com/paper/2411.00622/) (2024)
- [Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration](https://mlsanity.com/paper/2406.01422/) (2024)
- [CodeMonkeys: Scaling Test-Time Compute for Software Engineering](https://mlsanity.com/paper/2501.14723/) (2025)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [UniDebugger: Hierarchical Multi-Agent Framework for Unified Software Debugging](https://mlsanity.com/paper/2404.17153/) (2024)
- [CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification](https://mlsanity.com/paper/2405.00253/) (2024)
- [CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2408.13001/) (2024)
- [CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?](https://mlsanity.com/paper/2408.10718/) (2024)
- [SoRFT: Issue Resolving with Subtask-oriented Reinforced Fine-Tuning](https://mlsanity.com/paper/2502.20127/) (2025)
- [DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation](https://mlsanity.com/paper/2408.13204/) (2024)
- [ContrastRepair: Enhancing Conversation-Based Automated Program Repair via Contrastive Test Case Pairs](https://mlsanity.com/paper/2403.01971/) (2024)
- [Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?](https://mlsanity.com/paper/2410.01353/) (2024)
- [LPR: Large Language Models-Aided Program Reduction](https://mlsanity.com/paper/2312.13064/) (2023)
- [Enhancing Differential Testing With LLMs For Testing Deep Learning Libraries](https://mlsanity.com/paper/2406.07944/) (2024)

## Built on by (most cited first, 26 of 26)

- [SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents](https://mlsanity.com/paper/2505.20411/) (2025)
- [SWE-Exp: Experience-Driven Software Issue Resolution](https://mlsanity.com/paper/2507.23361/) (2025)
- [When Agents go Astray: Course-Correcting SWE Agents with PRMs](https://mlsanity.com/paper/2509.02360/) (2025)
- [Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs](https://mlsanity.com/paper/2506.19290/) (2025)
- [Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks](https://mlsanity.com/paper/2512.03262/) (2025)
- [Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering](https://mlsanity.com/paper/2505.23604/) (2025)
- [Generative AI Act II: Test Time Scaling Drives Cognition Engineering](https://mlsanity.com/paper/2504.13828/) (2025)
- [WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning](https://mlsanity.com/paper/2509.22644/) (2025)
- [Co-PatcheR: Collaborative Software Patching with Component(s)-specific Small Reasoning Models](https://mlsanity.com/paper/2505.18955/) (2025)
- [SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents](https://mlsanity.com/paper/2601.22129/) (2026)
- [Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoning](https://mlsanity.com/paper/2506.04611/) (2025)
- [EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models](https://mlsanity.com/paper/2506.08375/) (2025)
- [Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems](https://mlsanity.com/paper/2506.17208/) (2025)
- [Scaling Coding Agents via Atomic Skills](https://mlsanity.com/paper/2604.05013/) (2026)
- [How Inference Compute Shapes Frontier LLM Evaluation](https://mlsanity.com/paper/2606.17930/) (2026)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [Multi-agent Collaboration with State Management](https://mlsanity.com/paper/2605.20563/) (2026)
- [Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns](https://mlsanity.com/paper/2505.23474/) (2025)
- [InfCode-C++: Intent-Guided Semantic Retrieval and AST-Structured Search for C++ Issue Resolution](https://mlsanity.com/paper/2511.16005/) (2025)
- [From Historical Patches to Repair Plans: Outcome-Conditioned Reasoning for Repository-Level Program Repair](https://mlsanity.com/paper/2601.23257/) (2026)
- [DockSmith: Scaling Reliable Coding Environments via an Agentic Docker Builder](https://mlsanity.com/paper/2602.00592/) (2026)
- [TN-AutoRCA: Benchmark Construction and Agentic Framework for Self-Improving Alarm-Based Root Cause Analysis in Telecommunication Networks](https://mlsanity.com/paper/2507.18190/) (2025)
- [EGSS: Entropy-guided Stepwise Scaling for Reliable Software Engineering](https://mlsanity.com/paper/2602.05242/) (2026)
- [ReasonBridge: Efficient Reasoning Transfer from Closed to Open-Source Language Models](https://mlsanity.com/paper/2506.22865/) (2025)
- [Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling](https://mlsanity.com/paper/2605.11299/) (2026)
- [Towards Automatically Pruning Logging Code with Coding Agents: How Far Are We?](https://mlsanity.com/paper/2610.04716/) (2026)

## Abstract (arXiv)

Recent advancements in software engineering agents have demonstrated promising capabilities in automating program improvements. However, their reliance on closed-source or resource-intensive models introduces significant deployment challenges in private environments, prompting a critical question: textitHow can personally deployable open-source LLMs achieve comparable code reasoning performance? To this end, we propose a unified Test-Time Compute scaling framework that leverages increased inference-time computation instead of larger models. Our framework incorporates two complementary strategies: internal TTC and external TTC. Internally, we introduce a textitdevelopment-contextualized trajectory synthesis method leveraging real-world software repositories to bootstrap multi-stage reasoning processes, such as fault localization and patch generation. We further enhance trajectory quality through rejection sampling, rigorously evaluating trajectories along accuracy and complexity. Externally, we propose a novel textitdevelopment-process-based search strategy guided by reward models and execution verification. This approach enables targeted computational allocation at critical development decision points, overcoming limitations of existing "end-point only" verification methods. Evaluations on SWE-bench Verified demonstrate our 32B model achieves a 46% issue resolution rate, surpassing significantly larger models such as DeepSeek R1 671B and OpenAI o1. Additionally, we provide the empirical validation of the test-time scaling phenomenon within SWE agents, revealing that models dynamically allocate more tokens to increasingly challenging problems, effectively enhancing reasoning capabilities. We publicly release all training data, models, and code to facilitate future research. https://github.com/yingweima2022/SWE-Reasoner