mlsanityData to 7 Oct 2026

DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors

arXiv 2406.11427 ↗First posted 17 Jun 2024eess.AS cs.AI cs.CLcited by 31

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Large-scale latent diffusion models (LDMs) excel in content generation across various modalities, but their reliance on phonemes and durations in text-to-speech (TTS) limits scalability and access from other fields. While recent studies show potential in removing these domain-specific factors, performance remains suboptimal. In this work, we introduce DiTTo-TTS, a Diffusion Transformer (DiT)-based TTS model, to investigate whether LDM-based TTS can achieve state-of-the-art performance without domain-specific factors. Through rigorous analysis and empirical exploration, we find that (1) DiT with minimal modifications outperforms U-Net, (2) variable-length modeling with a speech length predictor significantly improves results over fixed-length approaches, and (3) conditions like semantic alignment in speech latent representations are key to further enhancement. By scaling our training data to 82K hours and the model size to 790M parameters, we achieve superior or comparable zero-shot performance to state-of-the-art TTS models in naturalness, intelligibility, and speaker similarity, all without relying on domain-specific factors. Speech samples are available at https://ditto-tts.github.io.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 73 of its 91 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

73 of 91 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models4 Jun
  2. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models5 Mar
  3. 2023Audiobox: Unified Audio Generation with Natural Language Prompts25 Dec
  4. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speech2 Nov
  5. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models13 Jun
  6. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  7. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  8. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance23 Nov
  9. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  10. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 63

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  13. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  14. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  15. 2018Hierarchical Neural Story Generationcited by 1,182
  16. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  17. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  18. 2022High Fidelity Neural Audio Compressioncited by 673
  19. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  20. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  21. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  22. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  23. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  24. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  25. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  26. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  27. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  28. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  29. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  30. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  31. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  32. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  33. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  34. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  35. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  36. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  37. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  38. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 11; 2026: 12.

31 papers.

Sort:

By area:

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  2. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  3. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 39
  4. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matchingcited by 30
  5. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  6. 2024StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matchingcited by 14
  7. 2024PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generationcited by 12
  8. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  9. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  10. 2024Efficient Generative Modeling with Residual Vector Quantization-Based Tokenscited by 7
  11. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusioncited by 6
  12. 2025DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generationcited by 5
19 more
  1. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Spacecited by 3
  2. 2025Length-Aware Rotary Position Embedding for Text-Speech Alignmentcited by 2
  3. 2026DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronizationcited by 2
  4. 2025SupertonicTTS: Towards Highly Efficient and Streamlined Text-to-Speech Systemcited by 1
  5. 2025Efficient Speech Language Modeling via Energy Distance in Continuous Latent Spacecited by 1
  6. 2025No Encore: Unlearning as Opt-Out in Music Generationcited by 1
  7. 2025UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Modelscited by 1
  8. 2026WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modelingcited by 1
  9. 2026Transcript-Free Flow-Matching Text-to-Speech via Speech Feature Conditioningcited by 1
  10. 2026DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speechcited by 1
  11. 2025Towards Lightweight and Stable Zero-shot TTS with Self-distilled Representation Disentanglementcited by 0
  12. 2025DiFlow-TTS: Compact and Low-Latency Zero-Shot Text-to-Speech with Discrete Flow Matchingcited by 0
  13. 2026SNAP: Speaker Nulling for Artifact Projection in Speech Deepfake Detectioncited by 0
  14. 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
  15. 2026Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Surveycited by 0
  16. 2026RobustSpeechFlow: Learning Robust Text-to-Speech Trajectories via Augmentation-based Contrastive Flow Matchingcited by 0
  17. 2026Mask, Sample, Revise: A Revisable CTMC Inference Stack for Guided Discrete Flow Matching Text-to-Speechcited by 0
  18. 2026Towards Real-world Environment-aware Zero-shot Text-to-speech Synthesis via Disentangled Audio Infillingcited by 0
  19. 2026Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffVoice: Text-to-Speech with Latent Diffusion
  2. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models
  3. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  4. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis
  5. 2026Raon-OpenTTS: Open Models and Data for Robust Text-to-Speech ↗
  6. 2024DPI-TTS: Directional Patch Interaction for Fast-Converging and Style Temporal Modeling in Text-to-Speech
  7. 2025M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis
  8. 2025Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment