mlsanityData to 7 Oct 2026

ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use

arXiv 2501.02506 ↗First posted 5 Jan 2025cs.CLcited by 48

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models (LLMs). However, progress has been hindered by a lack of reliable evaluation datasets. To address this, we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach that includes tool creation, document refinement, and code generation. We evaluate 14 LLMs across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT), uncovering significant challenges in handling multi-hop tool-use scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%, underscoring substantial room for improvement. Further analysis reveals variations in tool-use strategies for various families, offering actionable insights to guide the development of more effective approaches. Code and data can be found in https://huggingface.co/datasets/bytedance-research/ToolHop.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 27 of its 42 references that are in the map. First posted on arXiv 5 Jan 2025.

Builds on

27 of 42 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024ToolACE: Winning the Points of LLM Function Calling2 Sep
  2. 2024Tool Learning with Large Language Models: A Survey28 May
  3. 2024Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark14 May
  4. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages16 Feb
  5. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls6 Feb
  6. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning16 Jan
  7. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  9. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  10. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  11. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction30 May
  12. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
1 more
  1. 2023Tool Learning with Foundation Models17 Apr

Other references · 14

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  8. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  9. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  10. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138
  11. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 91
  12. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 85
2 more
  1. 2024MoreHopQA: More Than Multi-hop Reasoningcited by 37
  2. 2024CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 35.

48 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025DeepAgent: A General Reasoning Agent with Scalable Toolsetscited by 69
  3. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 68
  4. 2025Towards General Agentic Intelligence via Environment Scalingcited by 51
  5. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  6. 2026HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systemscited by 20
  7. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environmentscited by 19
  8. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  9. 2025M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmarkcited by 11
  10. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  11. 2025PhysToolBench: Benchmarking Physical Tool Understanding for MLLMscited by 9
  12. 2026SkillRAE: Agent Skill-Based Context Compilation for Retrieval-Augmented Executioncited by 8
36 more
  1. 2026CCTU: A Benchmark for Tool Use under Complex Constraintscited by 7
  2. 2025Hell or High Water: Evaluating Agentic Recovery from External Failurescited by 6
  3. 2026AutoAgent: Evolving Cognition and Elastic Memory Orchestration for Adaptive Agentscited by 5
  4. 2026Memory-Induced Tool-Drift in LLM Agentscited by 4
  5. 2026MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?cited by 4
  6. 2026AgentSelect: Benchmark for Narrative Query-to-Agent Recommendationcited by 3
  7. 2026UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agentscited by 3
  8. 2026PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agentscited by 3
  9. 2025Enhancing LLM Tool Use with High-quality Instruction Data from Knowledge Graphcited by 2
  10. 2025LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Modelscited by 2
  11. 2025RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learningcited by 2
  12. 2026Teaching LLMs to Learn Tool Trialing and Execution through Environment Interactioncited by 2
  13. 2026When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrievalcited by 2
  14. 2026DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agentscited by 2
  15. 2026What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agentscited by 2
  16. 2026PEARL: Plan Exploration and Adaptive Reinforcement Learning for Multihop Tool Usecited by 1
  17. 2026Enhancing Tool Calling in LLMs with the International Tool Calling Datasetcited by 1
  18. 2026Data-Driven Function Calling Improvements in Large Language Model for Online Financial QAcited by 1
  19. 2026The Amazing Agent Race: Strong Tool Users, Weak Navigatorscited by 1
  20. 2026SGA-MCTS: Decoupling Planning from Execution via Training-Free Atomic Experience Retrievalcited by 1
  21. 2026ToolWeave: Structured Synthesis of Complex Multi-Turn Tool-Calling Dialoguescited by 1
  22. 2026MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scalecited by 1
  23. 2025ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystemcited by 0
  24. 2026AgriAgent: Contract-Driven Planning and Capability-Aware Tool Orchestration in Real-World Agriculturecited by 0
  25. 2026Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarkscited by 0
  26. 2026If Only My CGM Could Speak: A Privacy-Preserving Agent for Question Answering over Continuous Glucose Datacited by 0
  27. 2026Only Say What You Know: Calibration-Aware Generation for Long-Form Factualitycited by 0
  28. 2026From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agentscited by 0
  29. 2026Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domaincited by 0
  30. 2026From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolutioncited by 0
  31. 2026CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agentscited by 0
  32. 2026SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Informationcited by 0
  33. 2026HiDiffTIR: Hierarchical Difficulty-Aware Policy Optimization for Multi-Turn Tool-Integrated Reasoningcited by 0
  34. 2026MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Modelscited by 0
  35. 2026RecToolBench: Benchmarking Recommendation-Specific Tool Orchestration under Fuzzy User Intentcited by 0
  36. 2026AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
  2. 2024ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph
  3. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
  4. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
  5. 2025ACEBench: Who Wins the Match Point in Tool Usage?
  6. 2026Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models
  7. 2025ToolCoder: A Systematic Code-Empowered Tool Learning Framework for Large Language Models
  8. 2023GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution