mlsanityData to 14 Oct 2026

DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models

arXiv 2502.19924 ↗First posted 27 Feb 2025cs.SD cs.AI eess.AScited by 2

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Conversational speech synthesis (CSS) aims to synthesize both contextually appropriate and expressive speech, and considerable efforts have been made to enhance the understanding of conversational context. However, existing CSS systems are limited to deterministic prediction, overlooking the diversity of potential responses. Moreover, they rarely employ language model (LM)-based TTS backbones, limiting the naturalness and quality of synthesized speech. To address these issues, in this paper, we propose DiffCSS, an innovative CSS framework that leverages diffusion models and an LM-based TTS backbone to generate diverse, expressive, and contextually coherent speech. A diffusion-based context-aware prosody predictor is proposed to sample diverse prosody embeddings conditioned on multimodal conversational context. Then a prosody-controllable LM-based TTS backbone is developed to synthesize high-quality speech with sampled prosody embeddings. Experimental results demonstrate that the synthesized speech from DiffCSS is more diverse, contextually coherent, and expressive than existing CSS systems

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 28 of its 32 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

28 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar

Other references · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  12. 2022High Fidelity Neural Audio Compressioncited by 687
15 more
  1. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  2. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  3. 2023Simple and Controllable Music Generationcited by 488
  4. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  5. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  6. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  7. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  8. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 101
  9. 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 69
  10. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  11. 2018On GANs and GMMscited by 62
  12. 2020Conversational End-to-End TTS for Voice Agentcited by 37
  13. 2024Generative Expressive Conversational Speech Synthesiscited by 16
  14. 2021Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modelingcited by 15
  15. 2023CONCSS: Contrastive-based Context Comprehension for Dialogue-appropriate Prosody in Conversational Speech Synthesiscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025NE-PADD: Leveraging Named Entity Knowledge for Robust Partial Audio Deepfake Detection via Attention Aggregationcited by 1
  2. 2026Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding
  2. 2023DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training
  3. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS
  4. 2023Diverse and Expressive Speech Prosody Prediction with Denoising Diffusion Probabilistic Model
  5. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusion
  6. 2024Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling
  7. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
  8. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models