# SWE-Shepherd: Advancing PRMs for Reinforcing Code Agents

arXiv: https://arxiv.org/abs/2604.10493 · HTML: https://mlsanity.com/paper/2604.10493/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 7 references that are in the map.
- First posted on arXiv 12 Apr 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 6 of 6)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution](https://mlsanity.com/paper/2502.18449/) (2025)
- [SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement](https://mlsanity.com/paper/2410.20285/) (2024)
- [Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards](https://mlsanity.com/paper/2506.11425/) (2025)
- [Web-Shepherd: Advancing PRMs for Reinforcing Web Agents](https://mlsanity.com/paper/2505.15277/) (2025)
- [SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs](https://mlsanity.com/paper/2504.14757/) (2025)

## Built on by (most cited first, 5 of 5)

- [CodeMidas: Scaling Agentic Coding RL Environments from Code Itself](https://mlsanity.com/paper/2609.22068/) (2026)
- [SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agents](https://mlsanity.com/paper/2609.32631/) (2026)
- [Opera: A Verbal Critic Framework for Long-horizon Coding Agents](https://mlsanity.com/paper/2609.33987/) (2026)
- [LatentSift: Policy-State Filtering for Token-Efficient Verification of Software Engineering Agents](https://mlsanity.com/paper/2609.36371/) (2026)
- [Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents](https://mlsanity.com/paper/2610.03634/) (2026)

## Abstract (arXiv)

Automating real-world software engineering tasks remains challenging for large language model (LLM)-based agents due to the need for long-horizon reasoning over large, evolving codebases and making consistent decisions across interdependent actions. Existing approaches typically rely on static prompting strategies or handcrafted heuristics to select actions such as code editing, file navigation, and test execution, but they lack fine-grained feedback on intermediate decisions. This leads to inefficient exploration, error propagation, and brittle solution trajectories. To address this limitation, we propose SWE-Shepherd, a framework that introduces Process Reward Models (PRMs) to provide dense, step-level supervision for repository-level code agents. Using trajectories from SWE-Bench, we construct an action-level reward dataset and train a lightweight reward model on a base LLM to estimate the usefulness of intermediate actions. During inference, the PRM evaluates candidate actions and guides the agent toward higher-reward decisions without requiring full reinforcement learning. Experiments on SWE-Bench Verified demonstrate improved interaction efficiency and action quality, while also highlighting challenges in aligning intermediate rewards with final task success.