Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Parallel coding agents can produce patches that work alone but fail when merged. This happens when one agent changes an interface or rule that another agent still relies on. We study these failures with stale, a benchmark for semantic coordination. Our evaluation runs the same tests on each patch alone and on their combination, counting only failures introduced by combining the patches. We use three tiers: synthetic tasks with controlled interface changes, pairs of merged pull requests, and constructed tasks that use real Django helpers. Among 834 runs on 417 mined Django pairs, only one showed interference after correcting the grading procedure. On constructed tasks using 12 Django helpers, interference occurred in 97% of runs. A message describing the completed concurrent change recovered 82% of runs. Reviewed pull requests may contain few unresolved parallel changes, even when agents fail on controlled tasks using real code. The constructed failure rates do not estimate how often these problems occur in practice.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 4 of its 17 references that are in the map. First posted on arXiv 21 Sep 2026.
Builds on
4 of 17 references.
Closest ancestors · 2
Same direction, newest first
- 2026AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub4 Apr
- 2024Training Software Engineering Agents and Verifiers with SWE-Gym30 Dec
Other references · 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025CodeCRDT: Observation-Driven Coordination for Multi-Agent LLM Code Generation
- 2026Verifying Coordination in Parallel Coding Agents: NP-Bench and a Scheduling Planner
- 2026EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
- 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
- 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
- 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
- 2026Confident and Wrong: Silent Semantic Failures in Coding Agents
- 2026The Specification Gap: Coordination Failure Under Partial Knowledge in Code Agents