mlsanityData to 14 Oct 2026

Closed-World Resolution Against Tool Hallucination in LLM Agents

arXiv 2609.19425 ↗First posted 16 Sep 2026cs.AI cs.CR cs.SEcited by 0

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

Tool-augmented large language model (LLM) agents fail in a way no tool-selection or tool-security method addresses: they call tools that do not exist and pass arguments no schema declares. Existing defenses either pick the right tool (selection) or constrain what an agent may do with real tools (gating), both of which presuppose the emitted call refers to a real tool at all. We show this is a structural blind spot: a hallucinated call is by construction not a decision any gate made, so no gate can reject it. This paper is primarily a measurement and benchmark study. We give a five-class taxonomy of tool hallucination (H1-H5) and, as a reference point, the Resolution Rung: a training-free, closed-world resolver (registry membership plus a signature check) whose interest is where it must sit, not what it computes. We prove hallucination defense must precede any causal gate, and characterize the one irreducible residue (borrowed arguments schema-indistinguishable from a valid call). Across ten hosted models under two invocation surfaces we measure 322 genuine hallucinations; fabricated-tool calls concentrate on the unconstrained raw-JSON surface (34 vs. 3), and model scale does not help (a 675B model matches a 7-8B one). We then extend to the Model Context Protocol, where merging several servers into one namespace creates hallucination surfaces a single registry cannot express (a second taxonomy, M1-M5); on the live MCP surface we measure 154 hallucinations, including from frontier models that were clean on the single-registry surface, because collisions and shadowing are structural to the merge. We release the versioned Hallucinated-Tools Benchmark (HTB) so any resolver is comparable across submissions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 12 of its 15 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

12 of 15 references.

References · 12

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  7. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  8. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 265
  9. 2023Efficient Guided Generation for Large Language Modelscited by 200
  10. 2025MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploitscited by 63
  11. 2025Enterprise-Grade Security for the Model Context Protocol (MCP): Frameworks and Mitigation Strategiescited by 42
  12. 2024ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Modelscited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding and Mitigating Hallucination Escape in Tool-Using LLM Agents
  2. 2026Internal Representations as Indicators of Hallucinations in Agent Tool Selection
  3. 2026Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents
  4. 2026Hallucination Cascade: Analyzing Error Propagation in Multi-Agent LLM Systems
  5. 2025MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them
  6. 2025Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systems
  7. 2026Refusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language Models ↗
  8. 2026AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification