mlsanityData to 7 Oct 2026

Towards human-like spoken dialogue generation between AI agents from written dialogue

arXiv 2310.01088 ↗First posted 2 Oct 2023cs.CL cs.LG cs.SDcited by 19

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The advent of large language models (LLMs) has made it possible to generate natural written dialogues between two agents. However, generating human-like spoken dialogues from these written dialogues remains challenging. Spoken dialogues have several unique characteristics: they frequently include backchannels and laughter, and the smoothness of turn-taking significantly influences the fluidity of conversation. This study proposes CHATS - CHatty Agents Text-to-Speech - a discrete token-based system designed to generate spoken dialogues based on written dialogues. Our system can generate speech for both the speaker side and the listener side simultaneously, using only the transcription from the speaker side, which eliminates the need for transcriptions of backchannels or laughter. Moreover, CHATS facilitates natural turn-taking; it determines the appropriate duration of silence after each utterance in the absence of overlap, and it initiates the generation of overlapping speech based on the phoneme sequence of the next utterance in case of overlap. Experimental evaluations indicate that CHATS outperforms the text-to-speech baseline, producing spoken dialogues that are more interactive and fluid while retaining clarity and intelligibility.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 29 of its 51 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

29 of 51 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  10. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  11. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  12. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
17 more
  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  2. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  3. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  4. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  5. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  6. 2022Generative Spoken Dialogue Language Modelingcited by 125
  7. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  8. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  9. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  10. 2020Conversational End-to-End TTS for Voice Agentcited by 37
  11. 2021Controllable Context-aware Conversational Speech Synthesiscited by 18
  12. 2021Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modelingcited by 15
  13. 2023M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesiscited by 6
  14. 2020Laughter Synthesis: Combining Seq2seq modeling with Transfer Learningcited by 5
  15. 2022End-to-End Text-to-Speech Based on Latent Representation of Speaking Styles Using Spontaneous Dialoguecited by 5
  16. 2023Laughter Synthesis using Pseudo Phonetic Tokens with a Large-scale In-the-wild Laughter Corpuscited by 5
  17. 2023A generative framework for conversational laughter: Its 'language model' and laughter sound synthesiscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 8; 2026: 4.

19 papers.

Sort:

By area:

  1. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  2. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  3. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  4. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  5. 2024Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reactioncited by 13
  6. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  7. 2024SPL: A Socratic Playground for Learning Powered by Large Language Modelcited by 9
  8. 2024Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generationcited by 7
  9. 2025ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matchingcited by 7
  10. 2024SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systemscited by 6
  11. 2025SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generationcited by 6
  12. 2025Speculative End-Turn Detector for Efficient Speech Chatbot Assistantcited by 5
7 more
  1. 2024J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modelingcited by 2
  2. 2025SpeechDialogueFactory: Generating High-Quality Speech Dialogue Data to Accelerate Your Speech-LLM Developmentcited by 1
  3. 2025TAVID: Text-Driven Audio-Visual Interactive Dialogue Generationcited by 1
  4. 2026Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Frameworkcited by 0
  5. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0
  6. 2026KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interactioncited by 0
  7. 2026Prompts versus Rules: Auditing and Controlling Speech Naturalness Behaviors in Voice User Simulatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents
  2. 2025Towards Human-like Multimodal Conversational Agent by Generating Engaging Speech
  3. 2026Agentic-DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech
  4. 2026VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
  5. 2025DialoSpeech: Dual-Speaker Dialogue Generation with LLM and Flow Matching
  6. 2024DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling
  7. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
  8. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction