mlsanityData to 14 Oct 2026

ToolTalk: Evaluating Tool-Usage in a Conversational Setting

arXiv 2311.10775 ↗First posted 15 Nov 2023cs.CL cs.AI cs.LGcited by 45

Not on the map. Why?

Cited by 35 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have displayed massive improvements in reasoning and decision-making skills and can hold natural conversations with users. Many recent works seek to augment LLM-based assistants with external tools so they can access private or up-to-date information and carry out actions on behalf of users. To better measure the performance of these assistants, this paper introduces ToolTalk, a benchmark consisting of complex user intents requiring multi-step tool usage specified through dialogue. ToolTalk contains 28 tools grouped into 7 plugins, and includes a complete simulated implementation of each tool, allowing for fully automated evaluation of assistants that rely on execution feedback. ToolTalk also emphasizes tools that externally affect the world rather than only tools for referencing or searching information. We evaluate GPT-3.5 and GPT-4 on ToolTalk resulting in success rates of 26% and 50% respectively. Our analysis of the errors reveals three major categories and suggests some future directions for improvement. We release ToolTalk at https://github.com/microsoft/ToolTalk.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

31 of 36 references.

References · 31

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  9. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  11. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  12. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
19 more
  1. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  2. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  3. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  4. 2023Tool Learning with Foundation Modelscited by 302
  5. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  6. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  7. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  8. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197
  9. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  10. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 140
  11. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 135
  12. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  13. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 84
  14. 2023Making Language Models Better Tool Learners with Execution Feedbackcited by 70
  15. 2020STAR: A Schema-Guided Dialog Dataset for Transfer Learningcited by 44
  16. 2020TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systemscited by 18
  17. 2022A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialogcited by 14
  18. 2022AnyTOD: A Programmable Task-Oriented Dialog Systemcited by 10
  19. 2019Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT qualitycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 14; 2026: 20.

45 papers.

Sort:

By area:

  1. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  2. 2024Tool Learning with Large Language Models: A Surveycited by 223
  3. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  4. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  5. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 76
  6. 2025VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applicationscited by 53
  7. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  8. 2024API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMscited by 39
  9. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  10. 2025ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Modelscited by 34
  11. 2024CACTUS: Chemistry Agent Connecting Tool-Usage to Sciencecited by 33
  12. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 23
33 more
  1. 2025VoiceAgentBench: Are Voice Assistants ready for agentic tasks?cited by 19
  2. 2025Non-Collaborative User Simulators for Tool Agentscited by 18
  3. 2026Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agentscited by 18
  4. 2024MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Modelscited by 17
  5. 2026ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agentscited by 14
  6. 2026ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Contextcited by 14
  7. 2026VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactionscited by 12
  8. 2025Chain-of-Tools: Utilizing Massive Unseen Tools in the CoT Reasoning of Frozen Language Modelscited by 10
  9. 2026TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarioscited by 10
  10. 2024FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogscited by 8
  11. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  12. 2024Sparse Rewards Can Self-Train Dialogue Agentscited by 6
  13. 2025LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedbackcited by 6
  14. 2025DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialoguescited by 6
  15. 2024Frontier AI Ethics: Anticipating and Evaluating the Societal Impacts of Language Model Agentscited by 5
  16. 2025FamilyTool: A Multi-hop Personalized Tool Use Benchmarkcited by 5
  17. 2025ToolCritic: Detecting and Correcting Tool-Use Errors in Dialogue Systemscited by 5
  18. 2026FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environmentscited by 4
  19. 2026STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamicscited by 4
  20. 2026Simulating Complex Multi-Turn Tool Calling Interactions in Stateless Execution Environmentscited by 3
  21. 2026CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environmentscited by 3
  22. 2026MemArbiter: Decision-Time Memory Arbitration for Long-Horizon LLM Agentscited by 3
  23. 2025When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialoguecited by 1
  24. 2026Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agentscited by 1
  25. 2026CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistantscited by 1
  26. 2025TAPS: Tool-Augmented Personalisation via Structured Taggingcited by 0
  27. 2025InData: Towards Secure Multi-Step, Tool-Based Data Analysiscited by 0
  28. 2026TOBench: A Task-Oriented Omni-Modal Benchmark for Real-World Tool-Using Agentscited by 0
  29. 2026AgentOmnia: Scaling Agentic Models for Full-Scenario Applicationscited by 0
  30. 2026Extracting Knowledge from Tools in LLM Agentscited by 0
  31. 2026MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Designcited by 0
  32. 2026Lost in the Request: How Communication Variation Disrupts Retrieval and Action in Email Agentscited by 0
  33. 2026RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agentscited by 0