mlsanityData to 7 Oct 2026

DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training

arXiv 2307.16549 ↗First posted 31 Jul 2023cs.SD cs.CL eess.AScited by 15

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Expressive text-to-speech systems have undergone significant advancements owing to prosody modeling, but conventional methods can still be improved. Traditional approaches have relied on the autoregressive method to predict the quantized prosody vector; however, it suffers from the issues of long-term dependency and slow inference. This study proposes a novel approach called DiffProsody in which expressive speech is synthesized using a diffusion-based latent prosody generator and prosody conditional adversarial training. Our findings confirm the effectiveness of our prosody generator in generating a prosody vector. Furthermore, our prosody conditional discriminator significantly improves the quality of the generated speech by accurately emulating prosody. We use denoising diffusion generative adversarial networks to improve the prosody generation speed. Consequently, DiffProsody is capable of generating prosody 16 times faster than the conventional diffusion model. The superior performance of our proposed method has been demonstrated via experiments.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 55 of its 79 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

55 of 79 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Models12 Jun
  2. 2023Diverse and Expressive Speech Prosody Prediction with Denoising Diffusion Probabilistic Model26 May
  3. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  12. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
38 more
  1. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  2. 2016Least Squares Generative Adversarial Networkscited by 1,000
  3. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  5. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  6. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  9. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  10. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  11. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  12. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  13. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  14. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  15. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  16. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  17. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  18. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  19. 2022Pushing the limits of raw waveform speaker recognitioncited by 49
  20. 2021PortaSpeech: Portable and High-Quality Generative Text-to-Speechcited by 47
  21. 2018Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesiscited by 31
  22. 2022Revisiting Over-Smoothness in Text to Speechcited by 31
  23. 2020Expressive TTS Training with Frame and Style Reconstruction Losscited by 30
  24. 2020Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesiscited by 28
  25. 2020Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesiscited by 27
  26. 2019Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representationscited by 23
  27. 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speechcited by 22
  28. 2018Sequence-to-Sequence Acoustic Modeling for Voice Conversioncited by 20
  29. 2022SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speechcited by 18
  30. 2021Phone-Level Prosody Modelling with GMM-Based MDN for Diverse and Controllable Speech Synthesiscited by 10
  31. 2022BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language modelcited by 10
  32. 2021Hierarchical prosody modeling and control in non-autoregressive parallel neural TTScited by 9
  33. 2021Emphasis control for parallel neural TTScited by 9
  34. 2022Language Model-Based Emotion Prediction Methods for Emotional Speech Synthesis Systemscited by 9
  35. 2022Simple and Effective Multi-sentence TTS with Expressive and Coherent Prosodycited by 5
  36. 2022ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in Paragraph-based TTScited by 5
  37. 2023PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modelingcited by 5
  38. 2023PITS: Variational Pitch Inference without Fundamental Frequency for End-to-End Pitch-controllable TTScited by 4

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 4; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  2. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 23
  3. 2024DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Datasetcited by 20
  4. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
  5. 2025FLowHigh: Towards Efficient and High-Quality Audio Super-Resolution with Single-Step Flow Matchingcited by 18
  6. 2024DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignmentcited by 8
  7. 2024DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Stylescited by 5
  8. 2024Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimizationcited by 4
  9. 2026SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generationcited by 4
  10. 2025Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokenscited by 2
  11. 2025VoiceCloak: A Multi-Dimensional Defense Framework against Unauthorized Diffusion-based Voice Cloningcited by 1
  12. 2025Curved Worlds, Clear Boundaries: Generalizing Speech Deepfake Detection using Hyperbolic and Spherical Geometry Spacescited by 1
3 more
  1. 2024Braille-to-Speech Generator: Audio Generation Based on Joint Fine-Tuning of CLIP and Fastspeech2cited by 0
  2. 2024CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generationcited by 0
  3. 2026Do Speech Emphasis Models Generalize across Languages and Emotions?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs
  2. 2025Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis
  3. 2023Adversarial Training of Denoising Diffusion Model Using Dual Discriminators for High-Fidelity Multi-Speaker TTS
  4. 2025DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models
  5. 2024Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling
  6. 2023Comparing normalizing flows and diffusion models for prosody and acoustic modelling in text-to-speech
  7. 2023Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion
  8. 2023DiffVoice: Text-to-Speech with Latent Diffusion