mlsanityData to 7 Oct 2026

E3 TTS: Easy End-to-End Diffusion-based Text to Speech

arXiv 2311.00945 ↗First posted 2 Nov 2023cs.SD cs.CL cs.LGcited by 30

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We propose Easy End-to-End Diffusion-based Text to Speech, a simple and efficient end-to-end text-to-speech model based on diffusion. E3 TTS directly takes plain text as input and generates an audio waveform through an iterative refinement process. Unlike many prior work, E3 TTS does not rely on any intermediate representations like spectrogram features or alignment information. Instead, E3 TTS models the temporal structure of the waveform through the diffusion process. Without relying on additional conditioning information, E3 TTS could support flexible latent structure within the given audio. This enables E3 TTS to be easily adapted for zero-shot tasks such as editing without any additional training. Experiments show that E3 TTS can generate high-fidelity audio, approaching the performance of a state-of-the-art neural TTS system. Audio samples are available at https://e3tts.github.io.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 35 of its 43 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

35 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023DiffVoice: Text-to-Speech with Latent Diffusion23 Apr
  2. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance23 Nov
  3. 2021EdiTTS: Score-based Editing for Controllable Text-to-Speech6 Oct
  4. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  10. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  11. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
18 more
  1. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  2. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  3. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  4. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  5. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  6. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  7. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
  8. 2018Efficient Neural Audio Synthesiscited by 218
  9. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  10. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
  11. 2020End-to-End Adversarial Text-to-Speechcited by 77
  12. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  13. 2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
  14. 2018FloWaveNet : A Generative Flow for Raw Audiocited by 54
  15. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  16. 2020Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesiscited by 38
  17. 2022SQuId: Measuring Speech Naturalness in Many Languagescited by 10
  18. 2021Personalized Keyphrase Detection using Speaker and Environment Informationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 11; 2026: 6.

30 papers.

Sort:

By area:

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  3. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  4. 2024Rolling Diffusion Modelscited by 102
  5. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 56
  6. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
  7. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  8. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 31
  9. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Modelscited by 23
  10. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  11. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  12. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
18 more
  1. 2025SpeakStream: Streaming Text-to-Speech with Interleaved Datacited by 8
  2. 2024E1 TTS: Simple and Fast Non-Autoregressive TTScited by 7
  3. 2025UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalitiescited by 5
  4. 2024An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTScited by 4
  5. 2025Voice Cloning: Comprehensive Surveycited by 4
  6. 2024Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representationcited by 3
  7. 2026Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesiscited by 3
  8. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Spacecited by 3
  9. 2024Multi-modal Adversarial Training for Zero-Shot Voice Cloningcited by 2
  10. 2024Facial Expression-Enhanced TTS: Combining Face Representation and Emotion Intensity for Adaptive Speechcited by 2
  11. 2025Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesiscited by 2
  12. 2026WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modelingcited by 1
  13. 2026BareWave: Waveform-Native Flow-Matching Text-to-Speechcited by 1
  14. 2026Bagpiper-TTS: Natural Language Guided Universal Speech Synthesiscited by 1
  15. 2025RingFormer: A Neural Vocoder with Ring Attention and Convolution-Augmented Transformercited by 0
  16. 2025Accelerating Diffusion Transformer-Based Text-to-Speech with Transformer Layer Cachingcited by 0
  17. 2025OLaPh: Optimal Language Phonemizercited by 0
  18. 2026Beyond Two-stage Diffusion TTS: Joint Structure and Content Refinement via Jump Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sample-Efficient Diffusion for Text-To-Speech Synthesis
  2. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  3. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
  4. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformer
  5. 2025ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling
  6. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
  7. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  8. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models