Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
When a tool-using agent's write times out or returns a server error, the action may already have taken effect. Retrying blindly duplicates it -- a second charge, a second announcement, a second deployment -- while giving up skips required work. We ask where exactly-once behaviour should be enforced: in the model, in the agent harness, or in the tool contract. We introduce LIMBO, a deterministic sandbox of six services with realistic contracts (optional idempotency keys, eventually consistent and missing read paths) and twelve fault modes injected at the service boundary, including late commits, redelivery and partial batches; every episode is graded against a ledger of committed effects. Across 25,930 episodes spanning nine recent models, three production agent harnesses, two contract variants and fifteen recovery conditions, the answer depends on the fault. When an immediate read-back can reveal what happened, the model decides: frontier models instructed to act exactly once almost never duplicate a write whose acknowledgement was lost (0.5%), weaker models often do, and the model explains 53% of the explained variance. When it cannot -- the request is still in flight, or the transport delivered it twice -- the same frontier models duplicate in 56% and 74% of episodes, and the contract explains 81%. We prove that no verification-only policy is exactly-once under late commits without a bound on in-flight time. Waiting works when such a bound is short and known, but with heavy-tailed in-flight delays even an hour of waiting per episode falls short of offering an idempotency key on every write, which lowers the duplicate rate from 28% to 4% because agents use keys when they exist. The harness barely matters, a guard that attaches keys transfers across harnesses unchanged, and agents reported success in 90% of the episodes in which they had duplicated an effect.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 47 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
30 of 47 references.
Closest ancestors · 5
Same direction, newest first
- 2026AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection7 Aug
- 2026Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems15 Jul
- 2026AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP13 Jul
- 2026ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions3 Jan
- 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents26 Jul
Other references · 25
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
- 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
- 2024AI Agents That Mattercited by 199
- 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
- 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
- 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
- 2025SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planningcited by 58
13 more
- 2026Towards a Science of AI Agent Reliabilitycited by 58
- 2025MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Usecited by 37
- 2026Atomix: Timely, Transactional Tool Use for Reliable Agentic Workflowscited by 27
- 2026The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Breakcited by 25
- 2025PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Casescited by 21
- 2026Cordon: Semantic Transactions for Tool-Using LLM Agentscited by 19
- 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agentscited by 15
- 2025WAREX: Web Agent Reliability Evaluation on Existing Benchmarkscited by 14
- 2026DART: Semantic Recoverability for Structured Tool Agentscited by 9
- 2025Hell or High Water: Evaluating Agentic Recovery from External Failurescited by 6
- 2026Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failurescited by 6
- 2026ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systemscited by 3
- 2026Agentic Transaction: Towards ACID-Compliant Agent Systemscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Tool Calls Succeed but Workflows Fail: Anomalies at the Agent-Tool Boundary
- 2026When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime
- 2025Dive into the Agent Matrix: A Realistic Evaluation of Self-Replication Risk in LLM Agents
- 2026How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines
- 2026Beyond Corrected Memory: Execution Consistency in Multi-Agent Systems
- 2026Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
- 2026The Troy Moment: How LLM Agents Adjudicate the Decision Point Under Impossible Tasks, Claimed Authority, and Peer Information
- 2026No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage