Executable Prefix Values for Locating Agent Failures
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Work on debugging LLM agents treats the "first mistake" marked by a human or LLM judge as ground truth for where a trajectory failed. We ask whether such labels point at causes or symptoms, and make the distinction executable: we restart the agent from reconstructed intermediate states of a failed trajectory and estimate its prefix value V(h_t), the probability of success from each step, with a certified point of no return (PNR) for a failure. On 150 SWE-bench Verified tasks with two frozen open-weight agents, we build such curves for 47 and 60 failed trajectories and probe every step labeled by four frontier judges under the Traverse rubric. We find that the labels do not mark what a controller needs. Judges of different lineages agree on the exact step in only 21/57 cases, and the newest judges flag mistakes in 8/10 and 10/10 successful runs. For 66% of the labels, the agent's success probability does not drop at the labeled step; only 7/188 answers name the decisive step. Restarting at the measured restart point raises expected success from 0.44 to 0.57 offline, and real restarts confirm the advantage, whereas restarting at a judge's label does not (-0.08 to -0.03). We release our code, curves, and labels.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 9 of its 29 references that are in the map. First posted on arXiv 13 Oct 2026.
Builds on
9 of 29 references.
Closest ancestors · 3
Same direction, newest first
- 2026Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks4 Aug
- 2025Where LLM Agents Fail and How They can Learn From Failures29 Sep
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
Other references · 6
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2026Locating Hidden Failures Makes Long-Horizon Agents More Reliablecited by 3
- 2026LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectoriescited by 2
- 2026Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regimecited by 1
- 2026Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoningcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Structural Baselines Matter for Agent Failure Detection and Attribution
- 2026TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
- 2026Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels
- 2026Real-Time Detection and Repair of LLM Agent Failures
- 2026AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
- 2026trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories
- 2025SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents
- 2026What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents