# Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking

arXiv: https://arxiv.org/abs/2203.03123 · HTML: https://mlsanity.com/paper/2203.03123/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking](https://mlsanity.com/paper/2005.02877/) (2020)
- [Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking](https://mlsanity.com/paper/1910.03544/) (2019)
- [Building a Conversational Agent Overnight with Dialogue Self-Play](https://mlsanity.com/paper/1801.04871/) (2018)
- [SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking](https://mlsanity.com/paper/1907.07421/) (2019)
- [Dialog State Tracking: A Neural Reading Comprehension Approach](https://mlsanity.com/paper/1908.01946/) (2019)
- [Global-Locally Self-Attentive Dialogue State Tracker](https://mlsanity.com/paper/1805.09655/) (2018)
- [A Sequence-to-Sequence Approach to Dialogue State Tracking](https://mlsanity.com/paper/2011.09553/) (2020)
- [Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answering](https://mlsanity.com/paper/1911.06192/) (2019)
- [BERT-DST: Scalable End-to-End Dialogue State Tracking with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1907.03040/) (2019)
- [Schema-Guided Dialogue State Tracking Task at DSTC8](https://mlsanity.com/paper/2002.01359/) (2020)
- [Efficient Context and Schema Fusion Networks for Multi-Domain Dialogue State Tracking](https://mlsanity.com/paper/2004.03386/) (2020)
- [Slot Self-Attentive Dialogue State Tracking](https://mlsanity.com/paper/2101.09374/) (2021)
- [Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation](https://mlsanity.com/paper/1909.00754/) (2019)
- [Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking](https://mlsanity.com/paper/2104.04466/) (2021)
- [Non-Autoregressive Dialog State Tracking](https://mlsanity.com/paper/2002.08024/) (2020)
- [HyST: A Hybrid Approach for Flexible and Accurate Dialogue State Tracking](https://mlsanity.com/paper/1907.00883/) (2019)

## Built on by (most cited first, 4 of 4)

- ["Do you follow me?": A Survey of Recent Approaches in Dialogue State Tracking](https://mlsanity.com/paper/2207.14627/) (2022)
- [Confidence Estimation for LLM-Based Dialogue State Tracking](https://mlsanity.com/paper/2409.09629/) (2024)
- [DSTEA: Improving Dialogue State Tracking via Entity Adaptive Pre-training](https://mlsanity.com/paper/2207.03858/) (2022)
- [Measuring Semantic Progress in Multi-turn Dialogue via Information Gain](https://mlsanity.com/paper/2606.12332/) (2026)

## Abstract (arXiv)

Dialogue state tracking (DST) aims to extract essential information from multi-turn dialogue situations and take appropriate actions. A belief state, one of the core pieces of information, refers to the subject and its specific content, and appears in the form of domain-slot-value. The trained model predicts "accumulated" belief states in every turn, and joint goal accuracy and slot accuracy are mainly used to evaluate the prediction; however, we specify that the current evaluation metrics have a critical limitation when evaluating belief states accumulated as the dialogue proceeds, especially in the most used MultiWOZ dataset. Additionally, we propose relative slot accuracy to complement existing metrics. Relative slot accuracy does not depend on the number of predefined slots, and allows intuitive evaluation by assigning relative scores according to the turn of each dialogue. This study also encourages not solely the reporting of joint goal accuracy, but also various complementary metrics in DST tasks for the sake of a realistic evaluation.