mlsanityData to 7 Oct 2026

On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models

arXiv 2402.12423 ↗First posted 19 Feb 2024cs.SD cs.CL cs.LGcited by 3

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

The incorporation of Denoising Diffusion Models (DDMs) in the Text-to-Speech (TTS) domain is rising, providing great value in synthesizing high quality speech. Although they exhibit impressive audio quality, the extent of their semantic capabilities is unknown, and controlling their synthesized speech's vocal properties remains a challenge. Inspired by recent advances in image synthesis, we explore the latent space of frozen TTS models, which is composed of the latent bottleneck activations of the DDM's denoiser. We identify that this space contains rich semantic information, and outline several novel methods for finding semantic directions within it, both supervised and unsupervised. We then demonstrate how these enable off-the-shelf audio editing, without any further training, architectural changes or data requirements. We present evidence of the semantic and acoustic qualities of the edited audio, and provide supplemental samples: https://latent-analysis-grad-tts.github.io/speech-samples/.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 19 of its 26 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

19 of 26 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  2. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech13 Jul
  3. 2022Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data30 May
  4. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis21 Apr
  5. 2022DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs28 Jan
  6. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  7. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech3 Apr
  8. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 11

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  6. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  7. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
  8. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
  9. 2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
  10. 2023Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Modelscited by 60
  11. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024Looks Too Good To Be True: An Information-Theoretic Analysis of Hallucinations in Generative Restoration Modelscited by 12
  2. 2024Anchored Diffusion for Video Face Reenactmentcited by 2
  3. 2024Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Studycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024PTQ4ADM: Post-Training Quantization for Efficient Text Conditional Audio Diffusion Models
  2. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
  3. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space
  4. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
  5. 2024Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model
  6. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  7. 2023A Study on Altering the Latent Space of Pretrained Text to Speech Models for Improved Expressiveness
  8. 2022BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis