# DialogueAgents: A Hybrid Agent-Based Speech Synthesis Framework for Multi-Party Dialogue

arXiv: https://arxiv.org/abs/2504.14482 · HTML: https://mlsanity.com/paper/2504.14482/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 20 Apr 2025.

## Builds on (most cited first, 15 of 15)

- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://mlsanity.com/paper/2402.12226/) (2024)
- [DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech](https://mlsanity.com/paper/2207.01063/) (2022)
- [WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark](https://mlsanity.com/paper/2406.05763/) (2024)
- [SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems](https://mlsanity.com/paper/2401.03945/) (2024)
- [ChatGPT-EDSS: Empathetic Dialogue Speech Synthesis Trained from ChatGPT-derived Context Word Embeddings](https://mlsanity.com/paper/2305.13724/) (2023)
- [Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data](https://mlsanity.com/paper/2412.01078/) (2024)

## Built on by (most cited first, 6 of 6)

- [ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning](https://mlsanity.com/paper/2506.09513/) (2025)
- [Towards Explicit Acoustic Evidence Perception in Audio LLMs for Speech Deepfake Detection](https://mlsanity.com/paper/2601.23066/) (2026)
- [Graph2Counsel: Clinically Grounded Synthetic Counseling Dialogue Generation from Client Psychological Graphs](https://mlsanity.com/paper/2604.20382/) (2026)
- [When Nobody Around Is Real: Exploring Public Opinions and User Experiences On the Multi-Agent AI Social Platform](https://mlsanity.com/paper/2601.18275/) (2026)
- [COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning](https://mlsanity.com/paper/2609.22697/) (2026)
- [RASSG: Repair-Aware Semantic Survival Gating for Streaming Tool Execution](https://mlsanity.com/paper/2610.14659/) (2026)

## Abstract (arXiv)

Speech synthesis is crucial for human-computer interaction, enabling natural and intuitive communication. However, existing datasets involve high construction costs due to manual annotation and suffer from limited character diversity, contextual scenarios, and emotional expressiveness. To address these issues, we propose DialogueAgents, a novel hybrid agent-based speech synthesis framework, which integrates three specialized agents -- a script writer, a speech synthesizer, and a dialogue critic -- to collaboratively generate dialogues. Grounded in a diverse character pool, the framework iteratively refines dialogue scripts and synthesizes speech based on speech review, boosting emotional expressiveness and paralinguistic features of the synthesized dialogues. Using DialogueAgent, we contribute MultiTalk, a bilingual, multi-party, multi-turn speech dialogue dataset covering diverse topics. Extensive experiments demonstrate the effectiveness of our framework and the high quality of the MultiTalk dataset. We release the dataset and code https://github.com/uirlx/DialogueAgents to facilitate future research on advanced speech synthesis models and customized data generation.