mlsanityData to 7 Oct 2026

τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

arXiv 2406.12045 ↗First posted 17 Jun 2024cs.AI cs.CLcited by 1,280

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose τ-bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state. We also propose a new metric (passk) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass8 <25% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably.

Key facts. Cited by 1,280 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 23 of its 31 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

23 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024ChatShop: Interactive Information Seeking with Language Agents15 Apr
  2. 2023Cognitive Architectures for Language Agents5 Sep
  3. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 20

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  11. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  12. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
8 more
  1. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 156
  2. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  3. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  4. 2021Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systemscited by 73
  5. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 45
  6. 2018User Modeling for Task Oriented Dialoguescited by 36
  7. 2023Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue Systemcited by 32
  8. 2024USimAgent: Large Language Models for Simulating Search Userscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 261; 2026: 1000.

1,280 papers.

Sort:

By area:

  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  2. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  3. 2025Kimi K2: Open Agentic Intelligencecited by 523
  4. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  5. 2025Humanity's Last Examcited by 467
  6. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  7. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  8. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  9. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  10. 2025Small Language Models are the Future of Agentic AIcited by 254
  11. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  12. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
38 more
  1. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  2. 2024AI Agents That Mattercited by 199
  3. 2025Survey on Evaluation of LLM-based Agentscited by 190
  4. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  5. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  6. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  7. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  9. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 118
  10. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 115
  11. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  12. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 107
  13. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  14. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  15. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  16. 2025Efficient Reasoning Models: A Surveycited by 93
  17. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 91
  18. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  19. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
  20. 2025Memp: Exploring Agent Procedural Memorycited by 88
  21. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  22. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
  23. 2025Agent Learning via Early Experiencecited by 79
  24. 2026Privileged Information Distillation for Language Modelscited by 79
  25. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 73
  26. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
  27. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  28. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environmentscited by 69
  29. 2026SkillX: Automatically Constructing Skill Knowledge Bases for Agentscited by 69
  30. 2025ACEBench: Who Wins the Match Point in Tool Usage?cited by 66
  31. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  32. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
  33. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 62
  34. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 61
  35. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 60
  36. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  37. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 58
  38. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57

50 most cited of 1,280 All 1,280 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Modeling User-System Behavior for Training-free Building of Private Domain Conversational Agents
  2. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
  3. 2026ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation
  4. 2026T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains
  5. 2025DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues
  6. 2024ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents
  7. 2026OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions
  8. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents