# Towards Detecting Inconsistencies in End-to-end Generated TODs

arXiv: https://arxiv.org/abs/2607.09338 · HTML: https://mlsanity.com/paper/2607.09338/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 10 Jul 2026.

## Builds on (most cited first, 12 of 12)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [Survey on Evaluation Methods for Dialogue Systems](https://mlsanity.com/paper/1905.04071/) (2019)
- [On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems](https://mlsanity.com/paper/1605.07669/) (2016)
- [Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models](https://mlsanity.com/paper/1709.06136/) (2017)
- [Effective Sequence-to-Sequence Dialogue State Tracking](https://mlsanity.com/paper/2108.13990/) (2021)
- [Towards Best Experiment Design for Evaluating Dialogue System Output](https://mlsanity.com/paper/1909.10122/) (2019)
- [Is this Dialogue Coherent? Learning from Dialogue Acts and Entities](https://mlsanity.com/paper/2006.10157/) (2020)
- [Coherence Models for Dialogue](https://mlsanity.com/paper/1806.08044/) (2018)
- [Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics](https://mlsanity.com/paper/2112.08321/) (2021)

## Abstract (arXiv)

Generative AI is profoundly transforming the core technologies behind conversational systems, shifting from component-based to end-to-end approaches. However, Large Language Models (LLMs) may still generate inconsistencies, a critical issue particularly in Task-Oriented Dialogues (TODs), where system responses must strictly adhere to information from a domain knowledge base (e.g., restaurants in a city). A single hallucination (e.g., suggesting a non-existent restaurant) can lead to severe task failures. We investigate a method for automatically detecting inconsistencies by conceptualizing TODs as a Constraint Satisfaction Problem (CSP), where variables represent dialogue segments referencing the conversational domain, and constraints among variables capture dialogue properties such as turn coherence and adherence to domain knowledge. We propose a pipeline that first identifies variables in a target dialogue and then applies a CSP solver to identify valid solutions. By comparing the target dialogue with valid variable assignments, we can detect inconsistencies and suggest minimal changes to ensure dialogue consistency. We demonstrate the high accuracy of the CSP-based approach in detecting inconsistencies, and provide a detailed analysis of our findings.