mlsanityData to 7 Oct 2026

Better speech synthesis through scaling

arXiv 2305.07243 ↗First posted 12 May 2023cs.SD cs.CL eess.AScited by 63

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In recent years, the field of image generation has been revolutionized by the application of autoregressive transformers and DDPMs. These approaches model the process of image generation as a step-wise probabilistic processes and leverage large amounts of compute and data to learn the image distribution. This methodology of improving performance need not be confined to images. This paper describes a way to apply advances in the image generative domain to speech synthesis. The result is TorToise -- an expressive, multi-voice text-to-speech system. All model code and trained weights have been open-sourced at https://github.com/neonbjb/tortoise-tts.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 7 of its 20 references that are in the map. First posted on arXiv 12 May 2023.

Builds on

7 of 20 references.

References · 7

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  7. 2021Hi-Fi Multi-Speaker English TTS Datasetcited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 23; 2025: 26; 2026: 11.

63 papers.

Sort:

By area:

  1. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  2. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  3. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  4. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  5. 2025Step-Audio 2 Technical Reportcited by 101
  6. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  7. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  8. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  9. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  10. 2025IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech Systemcited by 27
  11. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  12. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 19
38 more
  1. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  2. 2024I Can Hear You: Selective Robust Training for Deepfake Audio Detectioncited by 12
  3. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 10
  4. 2026MOSS-TTS Technical Reportcited by 10
  5. 2024Harder or Different? Understanding Generalization of Audio Deepfake Detectioncited by 9
  6. 2025GenVC: Self-Supervised Zero-Shot Voice Conversioncited by 8
  7. 2024Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representationscited by 7
  8. 2024Takin: A Cohort of Superior Quality Zero-shot Speech Generation Modelscited by 7
  9. 2025TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systemscited by 7
  10. 2024Small-E: Small Language Model with Linear Attention for Efficient Speech Synthesiscited by 6
  11. 2025Asynchronous Reasoning: Training-Free Interactive Thinking LLMscited by 6
  12. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  13. 2024VoiceWukong: Benchmarking Deepfake Voice Detectioncited by 5
  14. 2025SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesiscited by 5
  15. 2025Voice Cloning: Comprehensive Surveycited by 4
  16. 2023Custom Data Augmentation for low resource ASR using Bark and Retrieval-Based Voice Conversioncited by 3
  17. 2024PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoderscited by 3
  18. 2024Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with Multi-Modal Context and Large Language Modelcited by 3
  19. 2025Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancementcited by 3
  20. 2025TTS-1 Technical Reportcited by 3
  21. 2025Next Tokens Denoising for Speech Synthesiscited by 3
  22. 2026SonoEdit: Null-Space Constrained Knowledge Editing for Pronunciation Correction in LLM-Based TTScited by 3
  23. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Spacecited by 3
  24. 2024The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challengecited by 2
  25. 2025Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesiscited by 2
  26. 2025StreamFlow: Streaming Flow Matching with Block-wise Guided Attention Mask for Speech Token Decodingcited by 2
  27. 2025Investigating Stochastic Methods for Prosody Modeling in Speech Synthesiscited by 2
  28. 2025De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attackscited by 2
  29. 2025Multi-Modal Motion Retrieval by Learning a Fine-Grained Joint Embedding Spacecited by 2
  30. 2025Who Gets the Mic? Investigating Gender Bias in the Speaker Assignment of a Speech-LLMcited by 2
  31. 2025Gelina: Unified Speech and Gesture Synthesis via Interleaved Token Predictioncited by 2
  32. 2024Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Studycited by 1
  33. 2024Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesiscited by 1
  34. 2024The ISCSLP 2024 Conversational Voice Clone (CoVoC) Challenge: Tasks, Results and Findingscited by 1
  35. 2025DeePen: Penetration Testing for Audio Deepfake Detectioncited by 1
  36. 2025MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Promptcited by 1
  37. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementationcited by 1
  38. 2025Quantize More, Lose Less: Autoregressive Generation from Residually Quantized Speech Representationscited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Lightweight Pipeline for Noisy Speech Voice Cloning and Accurate Lip Sync Synthesis
  2. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis
  3. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
  4. 2023CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model
  5. 2026VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation
  6. 2023U-DiT TTS: U-Diffusion Vision Transformer for Text-to-Speech
  7. 2025MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis
  8. 2024Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis