# ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models

arXiv: https://arxiv.org/abs/2406.20015 · HTML: https://mlsanity.com/paper/2406.20015/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models](https://mlsanity.com/paper/2403.07714/) (2024)
- [Generating Benchmarks for Factuality Evaluation of Language Models](https://mlsanity.com/paper/2307.06908/) (2023)
- [EALM: Introducing Multidimensional Ethical Alignment in Conversational Information Retrieval](https://mlsanity.com/paper/2310.00970/) (2023)

## Built on by (most cited first, 25 of 25)

- [Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents](https://mlsanity.com/paper/2411.09523/) (2024)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning](https://mlsanity.com/paper/2509.25760/) (2025)
- [Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents](https://mlsanity.com/paper/2510.02837/) (2025)
- [AgentAbstain: Do LLM Agents Know When Not to Act?](https://mlsanity.com/paper/2607.10059/) (2026)
- [ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents](https://mlsanity.com/paper/2607.04686/) (2026)
- [Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?](https://mlsanity.com/paper/2605.19196/) (2026)
- [IP Leakage Attacks Targeting LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2505.12442/) (2025)
- [Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find](https://mlsanity.com/paper/2505.18148/) (2025)
- [Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use](https://mlsanity.com/paper/2605.14038/) (2026)
- [Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden States](https://mlsanity.com/paper/2608.02657/) (2026)
- [Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments](https://mlsanity.com/paper/2605.27209/) (2026)
- [Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents](https://mlsanity.com/paper/2605.09698/) (2026)
- [When Should LLMs Search? Counterfactual Supervision for Search Routing](https://mlsanity.com/paper/2607.05752/) (2026)
- [When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs](https://mlsanity.com/paper/2608.23623/) (2026)
- [Beyond Fluency: Toward Reliable Trajectories in Agentic IR](https://mlsanity.com/paper/2604.04269/) (2026)
- [Value-Guided Iterative Refinement and the DIQ-H Benchmark for Evaluating VLM Robustness](https://mlsanity.com/paper/2512.03992/) (2025)
- [Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows](https://mlsanity.com/paper/2605.24219/) (2026)
- [Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models](https://mlsanity.com/paper/2609.35732/) (2026)
- [Human-on-the-Loop Orchestration for AI-Assisted Legal Discovery](https://mlsanity.com/paper/2606.19812/) (2026)
- [CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents](https://mlsanity.com/paper/2607.14386/) (2026)
- [PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noise](https://mlsanity.com/paper/2608.02372/) (2026)
- [Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewards](https://mlsanity.com/paper/2608.07531/) (2026)
- [Robustness Analysis of Agentic AI to Inconsistent and Incomplete Tool Responses](https://mlsanity.com/paper/2608.22676/) (2026)
- [Closed-World Resolution Against Tool Hallucination in LLM Agents](https://mlsanity.com/paper/2609.19425/) (2026)

## Abstract (arXiv)

Tool-augmented large language models (LLMs) are rapidly being integrated into real-world applications. Due to the lack of benchmarks, the community has yet to fully understand the hallucination issues within these models. To address this challenge, we introduce a comprehensive diagnostic benchmark, ToolBH. Specifically, we assess the LLM's hallucinations through two perspectives: depth and breadth. In terms of depth, we propose a multi-level diagnostic process, including (1) solvability detection, (2) solution planning, and (3) missing-tool analysis. For breadth, we consider three scenarios based on the characteristics of the toolset: missing necessary tools, potential tools, and limited functionality tools. Furthermore, we developed seven tasks and collected 700 evaluation samples through multiple rounds of manual annotation. The results show the significant challenges presented by the ToolBH benchmark. The current advanced models Gemini-1.5-Pro and GPT-4o only achieve total scores of 45.3 and 37.0, respectively, on a scale of 100. In this benchmark, larger model parameters do not guarantee better performance; the training data and response strategies also play crucial roles in tool-enhanced LLM scenarios. Our diagnostic analysis indicates that the primary reason for model errors lies in assessing task solvability. Additionally, open-weight models suffer from performance drops with verbose replies, whereas proprietary models excel with longer reasoning.