# TD-EVAL: Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisons

arXiv: https://arxiv.org/abs/2504.19982 · HTML: https://mlsanity.com/paper/2504.19982/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](https://mlsanity.com/paper/2109.14739/) (2021)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://mlsanity.com/paper/1708.07149/) (2017)
- [MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation](https://mlsanity.com/paper/2104.00773/) (2021)
- [Are LLMs All You Need for Task-Oriented Dialogue?](https://mlsanity.com/paper/2304.06556/) (2023)
- [Shades of BLEU, Flavours of Success: The Case of MultiWOZ](https://mlsanity.com/paper/2106.05555/) (2021)
- [Structured Fusion Networks for Dialog](https://mlsanity.com/paper/1907.10016/) (2019)
- [Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances](https://mlsanity.com/paper/2106.02227/) (2021)
- [Large Language Models as Zero-shot Dialogue State Tracker through Function Calling](https://mlsanity.com/paper/2402.10466/) (2024)
- [LMUnit: Fine-grained Evaluation with Natural Language Unit Tests](https://mlsanity.com/paper/2412.13091/) (2024)
- [Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations](https://mlsanity.com/paper/2010.02495/) (2020)
- [A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions](https://mlsanity.com/paper/2504.16939/) (2025)
- [ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI](https://mlsanity.com/paper/2401.15471/) (2024)

## Built on by (most cited first, 10 of 10)

- [AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents](https://mlsanity.com/paper/2505.01592/) (2025)
- [Beyond Task-Oriented and Chitchat Dialogues: Proactive and Transition-Aware Conversational Agents](https://mlsanity.com/paper/2511.08835/) (2025)
- [MAC: A Multi-Agent Framework for Interactive User Clarification in Multi-turn Conversations](https://mlsanity.com/paper/2512.13154/) (2025)
- [ATOD: An Evaluation Framework and Benchmark for Agentic Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2601.11854/) (2026)
- [ESAinsTOD: A Unified End-to-End Schema-Aware Instruction-Tuning Framework for Task-Oriented Dialog Modeling](https://mlsanity.com/paper/2603.09691/) (2026)
- [RAG-DIVE: A Dynamic Approach for Multi-Turn Dialogue Evaluation in Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.16310/) (2026)
- [Clarification Is Not Enough: Post-Clarification Answering Remains the Bottleneck in Multi-Turn QA](https://mlsanity.com/paper/2605.25204/) (2026)
- [Measuring Semantic Progress in Multi-turn Dialogue via Information Gain](https://mlsanity.com/paper/2606.12332/) (2026)
- [TRACER: Early Failure Detection for Task-Oriented Dialogue](https://mlsanity.com/paper/2607.03974/) (2026)
- [DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions](https://mlsanity.com/paper/2609.18649/) (2026)

## Abstract (arXiv)

Task-oriented dialogue (TOD) systems are experiencing a revolution driven by Large Language Models (LLMs), yet the evaluation methodologies for these systems remain insufficient for their growing sophistication. While traditional automatic metrics effectively assessed earlier modular systems, they focus solely on the dialogue level and cannot detect critical intermediate errors that can arise during user-agent interactions. In this paper, we introduce TD-EVAL (Turn and Dialogue-level Evaluation), a two-step evaluation framework that unifies fine-grained turn-level analysis with holistic dialogue-level comparisons. At turn level, we evaluate each response along three TOD-specific dimensions: conversation cohesion, backend knowledge consistency, and policy compliance. Meanwhile, we design TOD Agent Arena that uses pairwise comparisons to provide a measure of dialogue-level quality. Through experiments on MultiWOZ 2.4 and τ-Bench, we demonstrate that TD-EVAL effectively identifies the conversational errors that conventional metrics miss. Furthermore, TD-EVAL exhibits better alignment with human judgments than traditional and LLM-based metrics. These findings demonstrate that TD-EVAL introduces a new paradigm for TOD system evaluation, efficiently assessing both turn and system levels with a plug-and-play framework for future research.