# CodeMonkeys: Scaling Test-Time Compute for Software Engineering

arXiv: https://arxiv.org/abs/2501.14723 · HTML: https://mlsanity.com/paper/2501.14723/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 1 of its 64 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 1 of 1)

- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)

## Built on by (most cited first, 30 of 51)

- [S*: Test Time Scaling for Code Generation](https://mlsanity.com/paper/2502.14382/) (2025)
- [SWE-Exp: Experience-Driven Software Issue Resolution](https://mlsanity.com/paper/2507.23361/) (2025)
- [RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models](https://mlsanity.com/paper/2506.17811/) (2025)
- [SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution](https://mlsanity.com/paper/2507.23348/) (2025)
- [Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use](https://mlsanity.com/paper/2504.04736/) (2025)
- [CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks](https://mlsanity.com/paper/2507.10535/) (2025)
- [Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute](https://mlsanity.com/paper/2503.23803/) (2025)
- [Otter: Generating Tests from Issues to Validate SWE Patches](https://mlsanity.com/paper/2502.05368/) (2025)
- [Agentic Rubrics as Contextual Verifiers for SWE Agents](https://mlsanity.com/paper/2601.04171/) (2026)
- [Verifier-free Test-Time Sampling for Vision-Language-Action Models](https://mlsanity.com/paper/2510.05681/) (2025)
- [Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach](https://mlsanity.com/paper/2512.02834/) (2025)
- [Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering](https://mlsanity.com/paper/2505.23604/) (2025)
- [Scaling Test-Time Compute for Agentic Coding](https://mlsanity.com/paper/2604.16529/) (2026)
- [Guiding Reasoning in Small Language Models with LLM Assistance](https://mlsanity.com/paper/2504.09923/) (2025)
- [Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents](https://mlsanity.com/paper/2602.07900/) (2026)
- [SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents](https://mlsanity.com/paper/2601.22129/) (2026)
- [Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agents](https://mlsanity.com/paper/2505.13652/) (2025)
- [Putting It All into Context: Simplifying Agents with LCLMs](https://mlsanity.com/paper/2505.08120/) (2025)
- [SeaView: Software Engineering Agent Visual Interface for Enhanced Workflow](https://mlsanity.com/paper/2504.08696/) (2025)
- [SuperCoder: Assembly Program Superoptimization with Large Language Models](https://mlsanity.com/paper/2505.11480/) (2025)
- [Investigating Test Overfitting on SWE-bench](https://mlsanity.com/paper/2511.16858/) (2025)
- [SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery](https://mlsanity.com/paper/2607.02807/) (2026)
- [J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge](https://mlsanity.com/paper/2505.11875/) (2025)
- [Pretraining Scaling Laws for Generative Evaluations of Language Models](https://mlsanity.com/paper/2509.24012/) (2025)
- [SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests](https://mlsanity.com/paper/2607.00990/) (2026)
- [Quokka: Accelerating Program Verification with LLMs via Invariant Synthesis](https://mlsanity.com/paper/2509.21629/) (2025)
- [LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?](https://mlsanity.com/paper/2510.09595/) (2025)
- [SciNav: A General Agent Framework for Scientific Coding Tasks](https://mlsanity.com/paper/2603.20256/) (2026)
- [Test-Time Scaling Makes Overtraining Compute-Optimal](https://mlsanity.com/paper/2604.01411/) (2026)
- [Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation](https://mlsanity.com/paper/2605.15669/) (2026)

## Abstract (arXiv)

Scaling test-time compute is a promising axis for improving LLM capabilities. However, test-time compute can be scaled in a variety of ways, and effectively combining different approaches remains an active area of research. Here, we explore this problem in the context of solving real-world GitHub issues from the SWE-bench dataset. Our system, named CodeMonkeys, allows models to iteratively edit a codebase by jointly generating and running a testing script alongside their draft edit. We sample many of these multi-turn trajectories for every issue to generate a collection of candidate edits. This approach lets us scale "serial" test-time compute by increasing the number of iterations per trajectory and "parallel" test-time compute by increasing the number of trajectories per problem. With parallel scaling, we can amortize up-front costs across multiple downstream samples, allowing us to identify relevant codebase context using the simple method of letting an LLM read every file. In order to select between candidate edits, we combine voting using model-generated tests with a final multi-turn trajectory dedicated to selection. Overall, CodeMonkeys resolves 57.4% of issues from SWE-bench Verified using a budget of approximately 2300 USD. Our selection method can also be used to combine candidates from different sources. Selecting over an ensemble of edits from existing top SWE-bench Verified submissions obtains a score of 66.2% and outperforms the best member of the ensemble on its own. We fully release our code and data at https://scalingintelligence.stanford.edu/pubs/codemonkeys.