mlsanityData to 14 Oct 2026

When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents

arXiv 2606.05806 ↗First posted 4 Jun 2026cs.AIcited by 17

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We introduce ToolMaze, a benchmark for dynamic path discovery and error recovery in TIR agents. To separate systematic replanning from blind trial-and-error, ToolMaze adopts a two-dimensional design: DAG-based topological complexity and a 2 × 2 taxonomy of tool perturbations (explicit/implicit, transient/permanent). Evaluations show that perturbations degrade performance across nearly all models, with the sharpest drops under implicit semantic failures. Driven by systemic over-trust in corrupted outputs, Perturbation Recovery Rate (PRR) plummets by around 37% in these scenarios, while complex topologies trap agents in futile trial-and-error loops. Crucially, agentic fault-tolerance improves with model scale 3.66× slower than basic task execution, highlighting dynamic replanning as a distinct bottleneck unaddressed by model scaling or prompting. Data and code are available at https://github.com/Zhudongsheng75/ToolMaze.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 25 of its 46 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

25 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Beyond ReAct: A Planner-Centric Framework for Complex Tool-Augmented LLM Reasoning13 Nov
  2. 2025More Vulnerable than You Think: On the Stability of Tool-Integrated LLM Agents27 Jun
  3. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun

Other references · 22

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  5. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  6. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  7. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  8. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  9. 2025Survey on Evaluation of LLM-based Agentscited by 204
  10. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  11. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
  12. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agentscited by 80
10 more
  1. 2024From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injectioncited by 38
  2. 2026ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditionscited by 36
  3. 2024ToolFlow: Boosting LLM Tool-Calling Through Natural and Coherent Dialogue Synthesiscited by 30
  4. 2025ToolTweak: An Attack on Tool Selection in LLM-based Agentscited by 26
  5. 2025PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Casescited by 22
  6. 2025Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missionscited by 12
  7. 2025Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning Systemcited by 10
  8. 2026Robust Tool Use via Fission-GRPO: Learning to Recover from Execution Errorscited by 10
  9. 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agentscited by 10
  10. 2026STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamicscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failurescited by 10
  2. 2026AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCPcited by 5
  3. 2026Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agentscited by 3
  4. 2026Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injectioncited by 2
  5. 2026Outcome Monitors: Recovery Affordances for Silent Tool Failurescited by 2
  6. 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluationcited by 2
  7. 2026MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Modelscited by 1
  8. 2026Engineering Reliable Commit Gates for Agentic AI: Cost-Aware Verification Portfolios under Common-Mode Data Failurescited by 1
  9. 2026OctoNest: Adaptive Cross-Device Execution through Stateful Controlcited by 0
  10. 2026ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agentscited by 0
  11. 2026Stale Does Not Mean Unsafe: Guard Precision for Tool-Using LLM Agents under Infrastructure State Racescited by 0
  12. 2026Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failurescited by 0
5 more
  1. 2026One Readout, Many Repairs: Diffusion-Guided Hierarchical Search for Tool-Agent Repaircited by 0
  2. 2026From Migration to Calibration: Aligning Agents with Deployment Requirementscited by 0
  3. 2026Quantization Effects on Tool-Failure Recovery Vary Across Prompts and Evaluation Designscited by 0
  4. 2026Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agentscited by 0
  5. 2026State Representation Shapes Long-Horizon Agent Learning and Transfer: An Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents
  2. 2026Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
  3. 2026ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems
  4. 2026PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
  5. 2026AgentEscapeBench: Evaluating Out-of-Domain Tool-Grounded Reasoning in LLM Agents
  6. 2026Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures
  7. 2026SiLR: Structure-Preserving Admission and Process Reward for LLM Tool Agents
  8. 2026AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation