# Why Synthetic Isn't Real Yet: A Diagnostic Framework for Contact Center Dialogue Generation

arXiv: https://arxiv.org/abs/2508.18210 · HTML: https://mlsanity.com/paper/2508.18210/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 61 references that are in the map.
- First posted on arXiv 25 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [PLACES: Prompting Language Models for Social Conversation Synthesis](https://mlsanity.com/paper/2302.03269/) (2023)
- [Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions](https://mlsanity.com/paper/2312.02913/) (2023)
- [ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks](https://mlsanity.com/paper/1803.01937/) (2018)
- [NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notes](https://mlsanity.com/paper/2310.15959/) (2023)
- [Synthetic Dialogue Dataset Generation using LLM Agents](https://mlsanity.com/paper/2401.17461/) (2024)
- [DiaSynth: Synthetic Dialogue Generation Framework for Low Resource Dialogue Applications](https://mlsanity.com/paper/2409.19020/) (2024)
- [Synthetic Patient-Physician Dialogue Generation from Clinical Notes Using LLM](https://mlsanity.com/paper/2408.06285/) (2024)
- [xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark](https://mlsanity.com/paper/2310.08958/) (2023)
- [AI Coach Assist: An Automated Approach for Call Recommendation in Contact Centers for Agent Coaching](https://mlsanity.com/paper/2305.17619/) (2023)
- [MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialogues](https://mlsanity.com/paper/2408.14418/) (2024)
- [We've had this conversation before: A Novel Approach to Measuring Dialog Similarity](https://mlsanity.com/paper/2110.05780/) (2021)
- [What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Study](https://mlsanity.com/paper/2109.09105/) (2021)
- [Entity-level Sentiment Analysis in Contact Center Telephone Conversations](https://mlsanity.com/paper/2210.13401/) (2022)
- [Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion](https://mlsanity.com/paper/2412.12710/) (2024)
- [Real-World En Call Center Transcripts Dataset with PII Redaction](https://mlsanity.com/paper/2507.02958/) (2025)
- [Conversations as a Source for Teaching Scientific Concepts at Different Education Levels](https://mlsanity.com/paper/2404.10475/) (2024)
- [SPADE: Structured Prompting Augmentation for Dialogue Enhancement in Machine-Generated Text Detection](https://mlsanity.com/paper/2503.15044/) (2025)

## Built on by (most cited first, 1 of 1)

- [Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios](https://mlsanity.com/paper/2601.15290/) (2025)

## Abstract (arXiv)

Synthetic data is increasingly critical for contact centers, where privacy constraints and data scarcity limit the availability of real conversations. However, generating synthetic dialogues that are realistic and useful for downstream applications remains challenging. In this work, we benchmark multiple generation strategies guided by structured supervision on call attributes (Intent Summaries, Topic Flows, and Quality Assurance (QA) Forms) across multiple languages. To test downstream utility, we evaluate synthetic transcripts on an automated quality assurance (AutoQA) task, finding that prompts optimized on real transcripts consistently outperform those optimized on synthetic transcripts. These results suggest that current synthetic transcripts fall short in capturing the full realism of real agent-customer interactions. To highlight these downstream gaps, we introduce a diagnostic evaluation framework comprising 17 metrics across four dimensions: (1) Emotional and Sentiment Arcs, (2) Linguistic Complexity, (3) Interaction Style, and (4) Conversational Properties. Our analysis shows that even with structured supervision, current generation strategies exhibit measurable deficiencies in sentiment fidelity, disfluency modeling, behavioral variation, and conversational realism. Together, these results highlight the importance of diagnostic, metric-driven evaluation for synthetic conversation generation intended for downstream applications.