# CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues

arXiv: https://arxiv.org/abs/2412.07515 · HTML: https://mlsanity.com/paper/2412.07515/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 57 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Builds on (most cited first, 10 of 10)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [Description-Driven Task-Oriented Dialog Modeling](https://mlsanity.com/paper/2201.08904/) (2022)
- [ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format](https://mlsanity.com/paper/2211.17148/) (2022)
- [Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluation](https://mlsanity.com/paper/2305.13857/) (2023)
- [A Chit-Chats Enhanced Task-Oriented Dialogue Corpora for Fuse-Motive Conversation Systems](https://mlsanity.com/paper/2205.05886/) (2022)

## Built on by (most cited first, 2 of 2)

- [MonoTODia: Translating Monologue Requests to Task-Oriented Dialogues](https://mlsanity.com/paper/2502.17268/) (2025)
- [APIDA-Chat: Structured Synthesis of API Search Dialogues to Bootstrap Conversational Agents](https://mlsanity.com/paper/2510.03743/) (2025)

## Abstract (arXiv)

Large-scale Wizard-Of-Oz dialogue datasets have enabled the training of deep learning-based dialogue systems. While they are successful as benchmark datasets, they lack certain types of utterances, which would make them more realistic. In this work, we investigate the creation of synthetic communication errors in an automatic pipeline. Based on linguistic theory, we propose and follow a simple error taxonomy. We focus on three types of miscommunications that could happen in real-world dialogues but are underrepresented in the benchmark dataset: misunderstandings, non-understandings and vaguely related questions. Our two-step approach uses a state-of-the-art Large Language Model (LLM) to first create the error and secondly the repairing utterance. We perform Language Model-based evaluation to ensure the quality of the generated utterances. We apply the method to the MultiWOZ dataset and evaluate it both qualitatively and empirically as well as with human judges. Our results indicate that current LLMs can aid in adding post-hoc miscommunications to benchmark datasets as a form of data augmentation. We publish the resulting dataset, in which nearly 1900 dialogues have been modified, as CoPrUS-MultiWOZ to facilitate future work on dialogue systems.