# Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances

arXiv: https://arxiv.org/abs/2204.03375 · HTML: https://mlsanity.com/paper/2204.03375/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 8 references that are in the map.
- First posted on arXiv 7 Apr 2022.

## Builds on (most cited first, 6 of 6)

- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking](https://mlsanity.com/paper/2005.02877/) (2020)
- [Efficient Dialogue State Tracking by Selectively Overwriting Memory](https://mlsanity.com/paper/1911.03906/) (2019)

## Built on by (most cited first, 14 of 14)

- [Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System](https://mlsanity.com/paper/2305.02468/) (2023)
- ["Do you follow me?": A Survey of Recent Approaches in Dialogue State Tracking](https://mlsanity.com/paper/2207.14627/) (2022)
- [OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Tracking](https://mlsanity.com/paper/2311.09758/) (2023)
- [Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations](https://mlsanity.com/paper/2305.14556/) (2023)
- [Interpreting User Requests in the Context of Natural Language Standing Instructions](https://mlsanity.com/paper/2311.09796/) (2023)
- [Confidence Estimation for LLM-Based Dialogue State Tracking](https://mlsanity.com/paper/2409.09629/) (2024)
- [Beyond Single-User Dialogue: Assessing Multi-User Dialogue State Tracking Capabilities of Large Language Models](https://mlsanity.com/paper/2506.10504/) (2025)
- [CorrectionLM: Self-Corrections with SLM for Dialogue State Tracking](https://mlsanity.com/paper/2410.18209/) (2024)
- [Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking](https://mlsanity.com/paper/2403.11123/) (2024)
- [Know Your Mistakes: Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling](https://mlsanity.com/paper/2501.10316/) (2025)
- [Hybrid Dialogue State Tracking for Persian Chatbots: A Language Model-Based Approach](https://mlsanity.com/paper/2510.01052/) (2025)
- [Faithfulness metric fusion: Improving the evaluation of LLM trustworthiness across domains](https://mlsanity.com/paper/2512.05700/) (2025)
- [Measuring Semantic Progress in Multi-turn Dialogue via Information Gain](https://mlsanity.com/paper/2606.12332/) (2026)
- [TPBench: A Turning-Point Benchmark for Dialogue Compression](https://mlsanity.com/paper/2610.02736/) (2026)

## Abstract (arXiv)

Dialogue State Tracking (DST) is primarily evaluated using Joint Goal Accuracy (JGA) defined as the fraction of turns where the ground-truth dialogue state exactly matches the prediction. Generally in DST, the dialogue state or belief state for a given turn contains all the intents shown by the user till that turn. Due to this cumulative nature of the belief state, it is difficult to get a correct prediction once a misprediction has occurred. Thus, although being a useful metric, it can be harsh at times and underestimate the true potential of a DST model. Moreover, an improvement in JGA can sometimes decrease the performance of turn-level or non-cumulative belief state prediction due to inconsistency in annotations. So, using JGA as the only metric for model selection may not be ideal for all scenarios. In this work, we discuss various evaluation metrics used for DST along with their shortcomings. To address the existing issues, we propose a new evaluation metric named Flexible Goal Accuracy (FGA). FGA is a generalized version of JGA. But unlike JGA, it tries to give penalized rewards to mispredictions that are locally correct i.e. the root cause of the error is an earlier turn. By doing so, FGA considers the performance of both cumulative and turn-level prediction flexibly and provides a better insight than the existing metrics. We also show that FGA is a better discriminator of DST model performance.