mlsanityData to 7 Oct 2026

ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities

arXiv 2408.04682 ↗First posted 8 Aug 2024cs.CL cs.AI cs.LGcited by 272

Not on the map. Why?

Cited by 191 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights into tool-use LLM capabilities. ToolSandbox evaluation framework is released at https://github.com/apple/ToolSandbox

Key facts. Cited by 272 papers in the map (data through 7 Oct 2026). Builds on 17 of its 29 references that are in the map. First posted on arXiv 8 Aug 2024.

Builds on

17 of 29 references.

References · 17

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Mistral 7Bcited by 3,065
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  8. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  9. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  11. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  12. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
5 more
  1. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  2. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  3. 2022A data-driven approach for learning to control computerscited by 87
  4. 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Gamescited by 24
  5. 2024GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applicationscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 67; 2026: 193.

272 papers.

Sort:

By area:

  1. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  2. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  3. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 257
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  5. 2025Survey on Evaluation of LLM-based Agentscited by 190
  6. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  7. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  8. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
  9. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  10. 2025ACEBench: Who Wins the Match Point in Tool Usage?cited by 66
  11. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  12. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
38 more
  1. 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenariocited by 53
  2. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  3. 2025Towards General Agentic Intelligence via Environment Scalingcited by 51
  4. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 51
  5. 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesiscited by 46
  6. 2025VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applicationscited by 45
  7. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 44
  8. 2025Infrastructure for AI Agentscited by 40
  9. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  10. 2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
  11. 2025When2Call: When (not) to Call Toolscited by 36
  12. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 35
  13. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
  14. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 32
  15. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  16. 2025ARE: Scaling Up Agent Environments and Evaluationscited by 28
  17. 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environmentscited by 28
  18. 2025Apple Intelligence Foundation Language Models: Tech Report 2025cited by 25
  19. 2024Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Basescited by 23
  20. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  21. 2026AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversationscited by 20
  22. 2024Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applicationscited by 19
  23. 2025HCAST: Human-Calibrated Autonomy Software Taskscited by 19
  24. 2025Goal Alignment in LLM-Based User Simulators for Conversational AIcited by 19
  25. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 19
  26. 2024Asynchronous LLM Function Callingcited by 18
  27. 2025On the Robustness of Agentic Function Callingcited by 17
  28. 2025Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systemscited by 17
  29. 2025CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactionscited by 16
  30. 2025MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Usecited by 16
  31. 2025Designing Domain-Specific Agents via Hierarchical Task Abstraction Mechanismcited by 16
  32. 2024Learning Evolving Tools for Large Language Modelscited by 15
  33. 2025Graph RAG-Tool Fusioncited by 15
  34. 2025LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queriescited by 15
  35. 2025Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systemscited by 15
  36. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  37. 2026SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfacescited by 15
  38. 2024CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planningcited by 14

50 most cited of 272 All 272 (JSON)