# ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use

arXiv: https://arxiv.org/abs/2501.02506 · HTML: https://mlsanity.com/paper/2501.02506/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 5 Jan 2025.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 27 of 27)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [ToolACE: Winning the Points of LLM Function Calling](https://mlsanity.com/paper/2409.00920/) (2024)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [TaskBench: Benchmarking Large Language Models for Task Automation](https://mlsanity.com/paper/2311.18760/) (2023)
- [AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls](https://mlsanity.com/paper/2402.04253/) (2024)
- [T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step](https://mlsanity.com/paper/2312.14033/) (2023)
- [Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios](https://mlsanity.com/paper/2401.17167/) (2024)
- [ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages](https://mlsanity.com/paper/2402.10753/) (2024)
- [Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark](https://mlsanity.com/paper/2405.08355/) (2024)
- [RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning](https://mlsanity.com/paper/2401.08326/) (2024)
- [MoreHopQA: More Than Multi-hop Reasoning](https://mlsanity.com/paper/2406.13397/) (2024)
- [CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs](https://mlsanity.com/paper/2408.02193/) (2024)

## Built on by (most cited first, 30 of 50)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.08755/) (2025)
- [DeepAgent: A General Reasoning Agent with Scalable Toolsets](https://mlsanity.com/paper/2510.21618/) (2025)
- [Towards General Agentic Intelligence via Environment Scaling](https://mlsanity.com/paper/2509.13311/) (2025)
- [ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces](https://mlsanity.com/paper/2604.05172/) (2026)
- [HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems](https://mlsanity.com/paper/2606.01779/) (2026)
- [Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments](https://mlsanity.com/paper/2508.08791/) (2025)
- [The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration](https://mlsanity.com/paper/2603.22862/) (2026)
- [Environment Scaling for Interactive Agentic Experience Collection: A Survey](https://mlsanity.com/paper/2511.09586/) (2025)
- [M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark](https://mlsanity.com/paper/2511.17729/) (2025)
- [Hell or High Water: Evaluating Agentic Recovery from External Failures](https://mlsanity.com/paper/2508.11027/) (2025)
- [PhysToolBench: Benchmarking Physical Tool Understanding for MLLMs](https://mlsanity.com/paper/2510.09507/) (2025)
- [SkillRAE: Agent Skill-Based Context Compilation for Retrieval-Augmented Execution](https://mlsanity.com/paper/2605.10114/) (2026)
- [CCTU: A Benchmark for Tool Use under Complex Constraints](https://mlsanity.com/paper/2603.15309/) (2026)
- [AutoAgent: Evolving Cognition and Elastic Memory Orchestration for Adaptive Agents](https://mlsanity.com/paper/2603.09716/) (2026)
- [MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?](https://mlsanity.com/paper/2606.23664/) (2026)
- [Memory-Induced Tool-Drift in LLM Agents](https://mlsanity.com/paper/2605.24941/) (2026)
- [Teaching LLMs to Learn Tool Trialing and Execution through Environment Interaction](https://mlsanity.com/paper/2601.12762/) (2026)
- [AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation](https://mlsanity.com/paper/2603.03761/) (2026)
- [UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents](https://mlsanity.com/paper/2604.11557/) (2026)
- [PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents](https://mlsanity.com/paper/2606.16215/) (2026)
- [What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents](https://mlsanity.com/paper/2608.27260/) (2026)
- [Enhancing LLM Tool Use with High-quality Instruction Data from Knowledge Graph](https://mlsanity.com/paper/2506.21071/) (2025)
- [LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models](https://mlsanity.com/paper/2508.05452/) (2025)
- [RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learning](https://mlsanity.com/paper/2601.00086/) (2025)
- [From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agents](https://mlsanity.com/paper/2606.20661/) (2026)
- [When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrieval](https://mlsanity.com/paper/2608.16502/) (2026)
- [DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents](https://mlsanity.com/paper/2608.18524/) (2026)
- [PEARL: Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Use](https://mlsanity.com/paper/2601.20439/) (2026)
- [Enhancing Tool Calling in LLMs with the International Tool Calling Dataset](https://mlsanity.com/paper/2603.05515/) (2026)

## Abstract (arXiv)

Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models (LLMs). However, progress has been hindered by a lack of reliable evaluation datasets. To address this, we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach that includes tool creation, document refinement, and code generation. We evaluate 14 LLMs across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT), uncovering significant challenges in handling multi-hop tool-use scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%, underscoring substantial room for improvement. Further analysis reveals variations in tool-use strategies for various families, offering actionable insights to guide the development of more effective approaches. Code and data can be found in https://huggingface.co/datasets/bytedance-research/ToolHop.