mlsanityData to 7 Oct 2026

Speech Synthesis From Continuous Features Using Per-Token Latent Diffusion

arXiv 2410.16048 ↗First posted 21 Oct 2024eess.AScited by 5

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We present SALAD, a zero-shot TTS autoregressive model operating over continuous speech representations. SALAD utilizes a per-token diffusion process to refine and predict continuous representations for the next time step. We compare our approach against a discrete variant of SALAD as well as publicly available zero-shot TTS systems, and conduct a comprehensive analysis of discrete versus continuous modeling techniques. Our results show that SALAD achieves superior intelligibility while matching the speech quality and speaker similarity of ground-truth audio.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 39 of its 49 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

39 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  2. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 36

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  11. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  12. 2022High Fidelity Neural Audio Compressioncited by 673
24 more
  1. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  2. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  3. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  4. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  5. 2023Simple and Controllable Music Generationcited by 480
  6. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  7. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  8. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  9. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  10. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  11. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  12. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  13. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  14. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  15. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  16. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  17. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  18. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  19. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  20. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  21. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  22. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  23. 2023Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognitioncited by 11
  24. 2024Low Bitrate High-Quality RVQGAN-based Discrete Speech Tokenizercited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Continuous Visual Autoregressive Generation via Score Maximizationcited by 9
  2. 2025Continuous Autoregressive Language Modelscited by 9
  3. 2025FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matchingcited by 8
  4. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 4
  5. 2026HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis
  2. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  3. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models
  4. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis
  5. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
  6. 2025DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation
  7. 2026DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis
  8. 2024Improving Robustness of Diffusion-Based Zero-Shot Speech Synthesis via Stable Formant Generation