Why Synthetic Isn't Real Yet: A Diagnostic Framework for Contact Center Dialogue Generation
Not on the map. Why?
Abstract (arXiv)
Synthetic data is increasingly critical for contact centers, where privacy constraints and data scarcity limit the availability of real conversations. However, generating synthetic dialogues that are realistic and useful for downstream applications remains challenging. In this work, we benchmark multiple generation strategies guided by structured supervision on call attributes (Intent Summaries, Topic Flows, and Quality Assurance (QA) Forms) across multiple languages. To test downstream utility, we evaluate synthetic transcripts on an automated quality assurance (AutoQA) task, finding that prompts optimized on real transcripts consistently outperform those optimized on synthetic transcripts. These results suggest that current synthetic transcripts fall short in capturing the full realism of real agent-customer interactions. To highlight these downstream gaps, we introduce a diagnostic evaluation framework comprising 17 metrics across four dimensions: (1) Emotional and Sentiment Arcs, (2) Linguistic Complexity, (3) Interaction Style, and (4) Conversational Properties. Our analysis shows that even with structured supervision, current generation strategies exhibit measurable deficiencies in sentiment fidelity, disfluency modeling, behavioral variation, and conversational realism. Together, these results highlight the importance of diagnostic, metric-driven evaluation for synthetic conversation generation intended for downstream applications.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 61 references that are in the map. First posted on arXiv 25 Aug 2025.
Builds on
22 of 61 references.
References · 22
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
- 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
- 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 168
- 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
- 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
- 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
- 2018ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Taskscited by 67
- 2023NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notescited by 48
- 2024Synthetic Dialogue Dataset Generation using LLM Agentscited by 39
- 2024DiaSynth: Synthetic Dialogue Generation Framework for Low Resource Dialogue Applicationscited by 18
- 2024Synthetic Patient-Physician Dialogue Generation from Clinical Notes Using LLMcited by 15
10 more
- 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
- 2023AI Coach Assist: An Automated Approach for Call Recommendation in Contact Centers for Agent Coachingcited by 8
- 2024MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialoguescited by 8
- 2021We've had this conversation before: A Novel Approach to Measuring Dialog Similaritycited by 7
- 2021What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Studycited by 5
- 2022Entity-level Sentiment Analysis in Contact Center Telephone Conversationscited by 4
- 2024Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertioncited by 3
- 2025Real-World En Call Center Transcripts Dataset with PII Redactioncited by 3
- 2024Conversations as a Source for Teaching Scientific Concepts at Different Education Levelscited by 1
- 2025SPADE: Structured Prompting Augmentation for Dialogue Enhancement in Machine-Generated Text Detectioncited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.