mlsanityData to 14 Oct 2026

Sample-Efficient Diffusion for Text-To-Speech Synthesis

arXiv 2409.03717 ↗First posted 1 Sep 2024cs.SD cs.AI cs.LGcited by 4

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

This work introduces Sample-Efficient Speech Diffusion (SESD), an algorithm for effective speech synthesis in modest data regimes through latent diffusion. It is based on a novel diffusion architecture, that we call U-Audio Transformer (U-AT), that efficiently scales to long sequences and operates in the latent space of a pre-trained audio autoencoder. Conditioned on character-aware language model representations, SESD achieves impressive results despite training on less than 1k hours of speech - far less than current state-of-the-art systems. In fact, it synthesizes more intelligible speech than the state-of-the-art auto-regressive model, VALL-E, while using less than 2% the training data.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 27 of its 32 references that are in the map. First posted on arXiv 1 Sep 2024.

Builds on

27 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  12. 2021Variational Diffusion Modelscited by 1,153
14 more
  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  2. 2022High Fidelity Neural Audio Compressioncited by 687
  3. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  4. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  5. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  7. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  8. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  9. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  11. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 304
  12. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  13. 2023On the Importance of Noise Scheduling for Diffusion Modelscited by 194
  14. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024E1 TTS: Simple and Fast Non-Autoregressive TTScited by 7
  2. 2025DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformerscited by 6
  3. 2025SpeechOp: Inference-Time Task Composition for Generative Speech Processingcited by 0
  4. 2026A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speechcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis
  2. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speech
  3. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  4. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models
  5. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
  6. 2025Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
  7. 2023U-DiT TTS: U-Diffusion Vision Transformer for Text-to-Speech
  8. 2023DiffVoice: Text-to-Speech with Latent Diffusion