mlsanityData to 7 Oct 2026

When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue

arXiv 2510.19186 ↗First posted 22 Oct 2025cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Evaluating conversational AI systems that use external tools is challenging, as errors can arise from complex interactions among user, agent, and tools. While existing evaluation methods assess either user satisfaction or agents' tool-calling capabilities, they fail to capture critical errors in multi-turn tool-augmented dialogues-such as when agents misinterpret tool results yet appear satisfactory to users. We introduce TRACE, a benchmark of systematically synthesized tool-augmented conversations covering diverse error cases. Evaluation with state-of-the-art conversation evaluation frameworks reveals that all approaches remain far from ideal performance, demonstrating the fundamental difficulty of this benchmark.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 16 of its 25 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

16 of 25 references.

References · 16

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  3. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  4. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  5. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  6. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  7. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  8. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 85
  9. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  10. 2021Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systemscited by 47
  11. 2023ToolTalk: Evaluating Tool-Usage in a Conversational Settingcited by 42
  12. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
4 more
  1. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  2. 2024Automated test generation to evaluate tool-augmented LLMs as conversational AI agentscited by 17
  3. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluationcited by 10
  4. 2025Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environmentscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025AgentChangeBench: A Multi-Dimensional Evaluation Framework for Goal-Shift Robustness in Conversational AI
  2. 2026ATOD: An Evaluation Framework and Benchmark for Agentic Task-Oriented Dialogue Systems
  3. 2026RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents
  4. 2026ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation
  5. 2026Agent Seer: Synthesizing Scenarios from Specification Understanding
  6. 2026Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
  7. 2025ToolCritic: Detecting and Correcting Tool-Use Errors in Dialogue Systems
  8. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains