mlsanityData to 7 Oct 2026

Conversational End-to-End TTS for Voice Agent

arXiv 2005.10438 ↗First posted 21 May 2020cs.SD eess.AScited by 37

Not on the map. Why?

Abstract (arXiv)

End-to-end neural TTS has achieved superior performance on reading style speech synthesis. However, it's still a challenge to build a high-quality conversational TTS due to the limitations of the corpus and modeling capability. This study aims at building a conversational TTS for a voice agent under sequence to sequence modeling framework. We firstly construct a spontaneous conversational speech corpus well designed for the voice agent with a new recording scheme ensuring both recording quality and conversational speaking style. Secondly, we propose a conversation context-aware end-to-end TTS approach which has an auxiliary encoder and a conversational context encoder to reinforce the information about the current utterance and its context in a conversation as well. Experimental results show that the proposed methods produce more natural prosody in accordance with the conversational context, with significant preference gains at both utterance-level and conversation-level. Moreover, we find that the model has the ability to express some spontaneous behaviors, like fillers and repeated words, which makes the conversational speaking style more realistic.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 12; 2024: 8; 2025: 5; 2026: 4.

37 papers.

Sort:

By area:

  1. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  2. 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 67
  3. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  4. 2023Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modelingcited by 17
  5. 2024Generative Expressive Conversational Speech Synthesiscited by 16
  6. 2021Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modelingcited by 15
  7. 2024Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reactioncited by 13
  8. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  9. 2022STUDIES: Corpus of Japanese Empathetic Dialogue Speech Towards Friendly Voice Agentcited by 9
  10. 2022FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesiscited by 9
  11. 2023CONCSS: Contrastive-based Context Comprehension for Dialogue-appropriate Prosody in Conversational Speech Synthesiscited by 8
  12. 2024Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Modelscited by 7
25 more
  1. 2025ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matchingcited by 7
  2. 2023M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesiscited by 6
  3. 2024Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modelingcited by 6
  4. 2022End-to-End Text-to-Speech Based on Latent Representation of Speaking Styles Using Spontaneous Dialoguecited by 5
  5. 2022ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in Paragraph-based TTScited by 5
  6. 2024Intra- and Inter-modal Context Interaction Modeling for Conversational Speech Synthesiscited by 5
  7. 2023ChatGPT-EDSS: Empathetic Dialogue Speech Synthesis Trained from ChatGPT-derived Context Word Embeddingscited by 4
  8. 2023Towards Spontaneous Style Modeling with Semi-supervised Pre-training for Conversational Text-to-Speech Synthesiscited by 4
  9. 2024Creating an African American-Sounding TTS: Guidelines, Technical Challenges,and Surprising Evaluationscited by 4
  10. 2025Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesiscited by 4
  11. 2022Acoustic Modeling for End-to-End Empathetic Dialogue Speech Synthesis Using Linguistic and Prosodic Contexts of Dialogue Historycited by 3
  12. 2023CALLS: Japanese Empathetic Dialogue Speech Corpus of Complaint Handling and Attentive Listening in Customer Centercited by 3
  13. 2024M3AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Datasetcited by 3
  14. 2024Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with Multi-Modal Context and Large Language Modelcited by 3
  15. 2023ContextSpeech: Expressive and Efficient Text-to-Speech for Paragraph Readingcited by 2
  16. 2025DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Modelscited by 2
  17. 2025Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesiscited by 2
  18. 2022Towards High-Quality Neural TTS for Low-Resource Languages by Learning Compact Speech Representationscited by 1
  19. 2023QS-TTS: Towards Semi-Supervised Text-to-Speech Synthesis via Vector-Quantized Self-Supervised Speech Representation Learningcited by 1
  20. 2023Large-Scale Automatic Audiobook Creationcited by 1
  21. 2026AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesiscited by 1
  22. 2023ComedicSpeech: Text To Speech For Stand-up Comedies in Low-Resource Scenarioscited by 0
  23. 2026Efficient ASR Training with Conversations that Never Happenedcited by 0
  24. 2026Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesiscited by 0
  25. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0