mlsanityData to 14 Oct 2026

A Study on Altering the Latent Space of Pretrained Text to Speech Models for Improved Expressiveness

arXiv 2311.10804 ↗First posted 17 Nov 2023cs.CL cs.AIcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

This report explores the challenge of enhancing expressiveness control in Text-to-Speech (TTS) models by augmenting a frozen pretrained model with a Diffusion Model that is conditioned on joint semantic audio/text embeddings. The paper identifies the challenges encountered when working with a VAE-based TTS model and evaluates different image-to-image methods for altering latent speech features. Our results offer valuable insights into the complexities of adding expressiveness control to TTS systems and open avenues for future research in this direction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 16 of its 20 references that are in the map. First posted on arXiv 17 Nov 2023.

Builds on

16 of 20 references.

References · 16

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  8. 2022Video Diffusion Modelscited by 2,305
  9. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  10. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  11. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  12. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
4 more
  1. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  2. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  3. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  4. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  2. 2025Text-Driven Voice Conversion via Latent State-Space Modeling ↗
  3. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS
  4. 2024Multimodal Latent Language Modeling with Next-Token Diffusion
  5. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
  6. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
  7. 2022Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencoder
  8. 2024LatentSpeech: Latent Diffusion for Text-To-Speech Generation