# Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation

arXiv: https://arxiv.org/abs/2005.07362 · HTML: https://mlsanity.com/paper/2005.07362/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 May 2020.

## Built on by (most cited first, 19 of 19)

- [Are LLMs All You Need for Task-Oriented Dialogue?](https://mlsanity.com/paper/2304.06556/) (2023)
- [Shades of BLEU, Flavours of Success: The Case of MultiWOZ](https://mlsanity.com/paper/2106.05555/) (2021)
- [A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning](https://mlsanity.com/paper/2202.13675/) (2022)
- ["Do you follow me?": A Survey of Recent Approaches in Dialogue State Tracking](https://mlsanity.com/paper/2207.14627/) (2022)
- [Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations](https://mlsanity.com/paper/2312.13871/) (2023)
- [SEAD: Self-Evolving Agent for Multi-Turn Service Dialogue](https://mlsanity.com/paper/2602.03548/) (2026)
- [Deciding Whether to Ask Clarifying Questions in Large-Scale Spoken Language Understanding](https://mlsanity.com/paper/2109.12451/) (2021)
- [Post-processing Networks: Method for Optimizing Pipeline Task-oriented Dialogue Systems using Reinforcement Learning](https://mlsanity.com/paper/2207.12185/) (2022)
- [Exploring ReAct Prompting for Task-Oriented Dialogue: Insights and Shortcomings](https://mlsanity.com/paper/2412.01262/) (2024)
- [How to Evaluate Your Dialogue Models: A Review of Approaches](https://mlsanity.com/paper/2108.01369/) (2021)
- [Joint System-Wise Optimization for Pipeline Goal-Oriented Dialog System](https://mlsanity.com/paper/2106.04835/) (2021)
- [Augmenting Slot Values and Contexts for Spoken Language Understanding with Pretrained Models](https://mlsanity.com/paper/2108.08451/) (2021)
- [A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots](https://mlsanity.com/paper/2111.01414/) (2021)
- [Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems](https://mlsanity.com/paper/2205.12554/) (2022)
- [Few-Shot Structured Policy Learning for Multi-Domain and Multi-Task Dialogues](https://mlsanity.com/paper/2302.11199/) (2023)
- [TA&AT: Enhancing Task-Oriented Dialog with Turn-Level Auxiliary Tasks and Action-Tree Based Scheduled Sampling](https://mlsanity.com/paper/2401.15626/) (2024)
- [PyTOD: Programmable Task-Oriented Dialogue with Execution Feedback](https://mlsanity.com/paper/2508.15456/) (2025)
- [Many Hands Make Light Work: Task-Oriented Dialogue System with Module-Based Mixture-of-Experts](https://mlsanity.com/paper/2405.09744/) (2024)
- [Dialogue Systems Engineering: A Survey and Future Directions](https://mlsanity.com/paper/2508.02279/) (2025)

## Abstract (arXiv)

There is a growing interest in developing goal-oriented dialog systems which serve users in accomplishing complex tasks through multi-turn conversations. Although many methods are devised to evaluate and improve the performance of individual dialog components, there is a lack of comprehensive empirical study on how different components contribute to the overall performance of a dialog system. In this paper, we perform a system-wise evaluation and present an empirical analysis on different types of dialog systems which are composed of different modules in different settings. Our results show that (1) a pipeline dialog system trained using fine-grained supervision signals at different component levels often obtains better performance than the systems that use joint or end-to-end models trained on coarse-grained labels, (2) component-wise, single-turn evaluation results are not always consistent with the overall performance of a dialog system, and (3) despite the discrepancy between simulators and human users, simulated evaluation is still a valid alternative to the costly human evaluation especially in the early stage of development.