mlsanityData to 7 Oct 2026

SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models

arXiv 2406.02328 ↗First posted 4 Jun 2024cs.SD eess.AScited by 23

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

In this study, we propose a simple and efficient Non-Autoregressive (NAR) text-to-speech (TTS) system based on diffusion, named SimpleSpeech. Its simpleness shows in three aspects: (1) It can be trained on the speech-only dataset, without any alignment information; (2) It directly takes plain text as input and generates speech through an NAR way; (3) It tries to model speech in a finite and compact latent space, which alleviates the modeling difficulty of diffusion. More specifically, we propose a novel speech codec model (SQ-Codec) with scalar quantization, SQ-Codec effectively maps the complex speech signal into a finite and compact latent space, named scalar latent space. Benefits from SQ-Codec, we apply a novel transformer diffusion model in the scalar latent space of SQ-Codec. We train SimpleSpeech on 4k hours of a speech-only dataset, it shows natural prosody and voice cloning ability. Compared with previous large-scale TTS models, it presents significant speech quality and generation speed improvement. Demos are released.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 38 of its 40 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

38 of 40 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speech2 Nov
  2. 2023DiffVoice: Text-to-Speech with Latent Diffusion23 Apr
  3. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  4. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan
  5. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS4 Nov
  6. 2022DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs28 Jan
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  10. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  11. 2016End-to-end Optimized Image Compressioncited by 673
  12. 2022High Fidelity Neural Audio Compressioncited by 673
19 more
  1. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  2. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  3. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  4. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  5. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  6. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  7. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  8. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  9. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
  10. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  11. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  12. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  13. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  14. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  15. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
  16. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  17. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  18. 2023ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understandingcited by 6
  19. 2024Pheme: Efficient and Conversational Speech Generationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 14; 2026: 1.

23 papers.

Sort:

By area:

  1. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  2. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 85
  3. 2024Scaling Transformers for Low-Bitrate High-Quality Speech Codingcited by 48
  4. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  5. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
  6. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  7. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 31
  8. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matchingcited by 30
  9. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  10. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  11. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  12. 2024E1 TTS: Simple and Fast Non-Autoregressive TTScited by 7
11 more
  1. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  2. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 6
  3. 2025DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Modelcited by 5
  4. 2025Voice Cloning: Comprehensive Surveycited by 4
  5. 2025VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editingcited by 3
  6. 2024Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesiscited by 1
  7. 2025SupertonicTTS: Towards Highly Efficient and Streamlined Text-to-Speech Systemcited by 1
  8. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementationcited by 1
  9. 2025Adaptive Duration Model for Text Speech Alignmentcited by 0
  10. 2025OLaPh: Optimal Language Phonemizercited by 0
  11. 2026Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunkingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
  2. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling
  3. 2024LatentSpeech: Latent Diffusion for Text-To-Speech Generation
  4. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis
  5. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space
  6. 2023Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding
  7. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
  8. 2024Sample-Efficient Diffusion for Text-To-Speech Synthesis