Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 11 of its 27 references that are in the map. First posted on arXiv 25 Mar 2026.
Builds on
11 of 27 references.
Closest ancestors · 4
Same direction, newest first
- 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?17 Nov
- 2025Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories31 Oct
- 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories23 Jun
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
Other references · 7
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
- 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
- 2026DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Modelscited by 8
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbationscited by 3
- 2026Code Isn't Memory: A Structural Codebase Index Inside a Coding Agentcited by 2
- 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agentscited by 1
- 2026The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Taskscited by 1
- 2026Analyzing and Mitigating Cost-Inefficient Behaviors in Coding Agentscited by 0
- 2026Analyzing and Mitigating Cost-Inefficient Behaviors in Coding Agents25 Sep 2026
- 2026The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks17 Aug 2026
- 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents7 Aug 2026
- 2026Code Isn't Memory: A Structural Codebase Index Inside a Coding Agent21 Jun 2026
- 2026RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations25 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
- 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
- 2026SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle
- 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
- 2026SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
- 2026TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
- 2025Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair
- 2026TACT: Mitigating Overthinking and Overacting in Coding Agents via Activation Steering