mlsanityData to 14 Oct 2026

DiffVoice: Text-to-Speech with Latent Diffusion

arXiv 2304.11750 ↗First posted 23 Apr 2023eess.AS cs.AI cs.HCcited by 13

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In this work, we present DiffVoice, a novel text-to-speech model based on latent diffusion. We propose to first encode speech signals into a phoneme-rate latent representation with a variational autoencoder enhanced by adversarial training, and then jointly model the duration and the latent representation with a diffusion model. Subjective evaluations on LJSpeech and LibriTTS datasets demonstrate that our method beats the best publicly available systems in naturalness. By adopting recent generative inverse problem solving algorithms for diffusion models, DiffVoice achieves the state-of-the-art performance in text-based speech editing, and zero-shot adaptation.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 32 of its 38 references that are in the map. First posted on arXiv 23 Apr 2023.

Builds on

32 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Solving Audio Inverse Problems with a Diffusion Model27 Oct
  2. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance23 Nov
  3. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  4. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech3 Apr

Other references · 28

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,416
  6. 2018Self-Attention Generative Adversarial Networkscited by 809
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  9. 2021Score-based Generative Modeling in Latent Spacecited by 636
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  11. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  12. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 508
16 more
  1. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  2. 2015A note on the evaluation of generative modelscited by 292
  3. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  4. 2021A Survey on Neural Speech Synthesiscited by 184
  5. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  6. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  7. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  8. 2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
  9. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40
  10. 2021EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusioncited by 32
  11. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  12. 2021GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesiscited by 18
  13. 2021CTC Variations Through New WFST Topologiescited by 17
  14. 2022SpeechPainter: Text-conditioned Speech Inpaintingcited by 17
  15. 2022CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech Editingcited by 16
  16. 2022RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 2; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matchingcited by 49
  2. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  3. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speechcited by 30
  4. 2023Federated Generative Learning with Foundation Modelscited by 23
  5. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Modelscited by 23
  6. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  7. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  8. 2024DLPO: Diffusion Model Loss-Guided Reinforcement Learning for Fine-Tuning Text-to-Speech Diffusion Modelscited by 7
  9. 2024E1 TTS: Simple and Fast Non-Autoregressive TTScited by 7
  10. 2025Voice Cloning: Comprehensive Surveycited by 4
  11. 2024Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRIcited by 2
  12. 2025InfiniteAudio: Infinite-Length Audio Generation with Consistencycited by 0
1 more
  1. 2026TCAM-Diff: Triplane-Aware Cross-Attention Medical Diffusion Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
  2. 2022DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs
  3. 2024LatentSpeech: Latent Diffusion for Text-To-Speech Generation
  4. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  5. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  6. 2023DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation
  7. 2024FastVoiceGrad: One-step Diffusion-Based Voice Conversion with Adversarial Conditional Diffusion Distillation
  8. 2023Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding