Outcome Monitors: Recovery Affordances for Silent Tool Failures
Not on the map. Why?
Abstract (arXiv)
When a tool call times out, the agent sees the failure and can route around it. A cached error page or negative price can instead arrive in the expected format and be consumed as fact. We introduce Outcome Monitors, which detect violations of outcome contracts mined from task-disjoint traces or derived from public schemas. On a violation, the monitor preserves the result and issues a nonbinding receipt naming the violated property and public recovery tools. In frozen, prespecified evaluations with injected failures, Outcome Monitors raise ToolMaze completion from 10.9% to 28.1% across four models in two provider families and replicate in a third. In tau-bench retail, completion improves by 14.0 and 12.0 points on two tiers. In separate ToolMaze controls, removing the recovery-tool list eliminates the measured gain and restoring it recovers the effect; diagnostic detail and timing produce no detectable differences. Gains concentrate where the fault blocks completion. On a suite transcribed from a published incident taxonomy, detection outside the mined vocabulary falls to 46%, though delivery continues and completion is unchanged. Recovery tools are the active receipt content in these controls; extending detection beyond the contract vocabulary remains open.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 19 Aug 2026.
Builds on
19 of 34 references.
References · 19
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Let's Verify Step by Stepcited by 3,598
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
- 2017Safe Reinforcement Learning via Shieldingcited by 427
- 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
- 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
- 2025SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planningcited by 58
- 2025PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Casescited by 21
7 more
- 2026Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agentscited by 18
- 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agentscited by 15
- 2026ToolGate: Contract-Grounded and Verified Tool Execution for LLMscited by 13
- 2025Agent-SAMA: State-Aware Mobile Assistantcited by 12
- 2026Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agentscited by 8
- 2026Reinforced Agent: Inference-Time Feedback for Tool-Calling Agentscited by 6
- 2026Contract2Tool: Learning Preconditions and Effects for Reliable Tool-Augmented LLM Agentscited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.