mlsanityData to 7 Oct 2026

Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code

arXiv 2603.24631 ↗First posted 25 Mar 2026cs.SE cs.AIcited by 5

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Code agents resolve 65-70% of SWE-bench Verified issues, but Pass@1 cannot tell us why the rest fail, and, as we show, capable-model failures are systematically misdiagnosed without trajectory data. We introduce TRAJEVAL, a training-free decomposition of agent trajectories into reference-patch-aligned search, read, and edit stages, and apply it across 16,758 trajectories spanning three architectures and seven models. The dominant failure of capable models is not localization: 60-69% of failures on SWE-Agent and OpenHands reach and edit the correct functions yet still produce incorrect patches, and the pattern persists for most models on the bash-only LiveSWEAgent. Within this Edit-Quality residual, we identify Coherence Collapse, where the agent reaches correct code and then overwrites or thrashes it, as the largest theme, replicating across SWE-bench Verified and the multilingual PolyBench Verified. In 5 cases, the agent produces a patch bit-identical to the gold reference mid-trajectory and destroys it later; an edit-commit checkpoint recovers all 5 against the SWE-bench Docker harness. A reference-free consensus-driven variant yields a directional +3.0 pp Pass@1 measurement on GPT-5 (p=0.08).

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 11 of its 27 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

11 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?17 Nov
  2. 2025Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories31 Oct
  3. 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories23 Jun
  4. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul

Other references · 7

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  7. 2026DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
  2. 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
  3. 2026SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle
  4. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
  5. 2026SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
  6. 2026TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
  7. 2025Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair
  8. 2026TACT: Mitigating Overthinking and Overacting in Coding Agents via Activation Steering