mlsanityData to 7 Oct 2026

τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment

arXiv 2506.07982 ↗First posted 9 Jun 2025cs.AI cs.CLcited by 558

Not on the map. Why?

Cited by 362 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing benchmarks for conversational AI agents simulate single-control environments, where only the AI agent can use tools to interact with the world, while the user remains a passive information provider. This differs from real-world scenarios like technical support, where users need to actively participate in modifying the state of the (shared) world. In order to address this gap, we introduce τ2-bench, with four key contributions: 1) A novel Telecom dual-control domain modeled as a Dec-POMDP, where both agent and user make use of tools to act in a shared, dynamic environment that tests both agent coordination and communication, 2) A compositional task generator that programmatically creates diverse, verifiable tasks from atomic components, ensuring domain coverage and controlled complexity, 3) A reliable user simulator tightly coupled with the environment, whose behavior is constrained by tools and observable states, improving simulation fidelity, 4) Fine-grained analysis of agent performance through multiple ablations including separating errors arising from reasoning vs communication/coordination. In particular, our experiments show significant performance drops when agents shift from no-user to dual-control, highlighting the challenges of guiding users. Overall, τ2-bench provides a controlled testbed for agents that must both reason effectively and guide user actions.

Key facts. Cited by 558 papers in the map (data through 7 Oct 2026). Builds on 19 of its 35 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

19 of 35 references.

References · 19

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  7. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  8. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  9. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  10. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  11. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 169
  12. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
7 more
  1. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 156
  2. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  3. 2021Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systemscited by 73
  4. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agentscited by 39
  5. 2018User Modeling for Task Oriented Dialoguescited by 36
  6. 2023Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue Systemcited by 32
  7. 2025IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systemscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 77; 2026: 481.

558 papers.

Sort:

By area:

  1. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  2. 2025Kimi K2: Open Agentic Intelligencecited by 523
  3. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  5. 2026Gemma 4 Technical Reportcited by 187
  6. 2026MiMo-V2-Flash Technical Reportcited by 163
  7. 2026Qwen3.5-Omni Technical Reportcited by 146
  8. 2025Towards a Science of Scaling Agent Systemscited by 125
  9. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  10. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  11. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  12. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environmentscited by 69
38 more
  1. 2026SkillX: Automatically Constructing Skill Knowledge Bases for Agentscited by 69
  2. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 61
  3. 2025LongCat-Flash Technical Reportcited by 54
  4. 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectoriescited by 53
  5. 2025Simulating Environments with Reasoning Models for Agent Trainingcited by 52
  6. 2025Towards General Agentic Intelligence via Environment Scalingcited by 51
  7. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 51
  8. 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligencecited by 50
  9. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  10. 2026MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Serverscited by 46
  11. 2025VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applicationscited by 45
  12. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  13. 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraintscited by 43
  14. 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 42
  15. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 40
  16. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  17. 2026LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routingcited by 35
  18. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 35
  19. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
  20. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  21. 2026MiniMax Sparse Attentioncited by 34
  22. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  23. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 32
  24. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 32
  25. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 32
  26. 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 31
  27. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  28. 2025MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool usecited by 30
  29. 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domainscited by 30
  30. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  31. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  32. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 28
  33. 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environmentscited by 28
  34. 2026AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agentscited by 28
  35. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 27
  36. 2025ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Datasetcited by 27
  37. 2026AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contextscited by 26
  38. 2025LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarioscited by 25

50 most cited of 558 All 558 (JSON)