mlsanityData to 14 Oct 2026

Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding

arXiv 2410.21951 ↗First posted 29 Oct 2024eess.AS cs.AI cs.SDcited by 17

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

The auto-regressive architecture, like GPTs, is widely used in modern Text-to-Speech (TTS) systems. However, it incurs substantial inference time, particularly due to the challenges in the next-token prediction posed by lengthy sequences of speech tokens. In this work, we introduce VADUSA, one of the first approaches to accelerate auto-regressive TTS through speculative decoding. Our results show that VADUSA not only significantly improves inference speed but also enhances performance by incorporating draft heads to predict future speech content auto-regressively. Furthermore, the inclusion of a tolerance mechanism during sampling accelerates inference without compromising quality. Our approach demonstrates strong generalization across large datasets and various types of speech tokens.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 29 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

28 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  9. 2022High Fidelity Neural Audio Compressioncited by 687
  10. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  11. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  12. 2018Efficient Neural Audio Synthesiscited by 219
14 more
  1. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  2. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  3. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  4. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
  5. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  6. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  7. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  8. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  9. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  10. 2024VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speechcited by 23
  11. 2024Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generationcited by 23
  12. 2023Fewer-token Neural Speech Codec with Time-invariant Codescited by 22
  13. 2023Acoustic BPE for Speech Generation with Discrete Tokenscited by 15
  14. 2024On the Effectiveness of Acoustic BPE in Decoder-Only TTScited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 8.

17 papers.

Sort:

By area:

  1. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  2. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  3. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 14
  4. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  5. 2025Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decodingcited by 6
  6. 2025Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Ratecited by 4
  7. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  8. 2026VoxServe: Streaming-Centric Serving System for Speech Language Modelscited by 3
  9. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  10. 2025SyncSpeech: Efficient and Low-Latency Text-to-Speech based on Temporal Masked Transformercited by 2
  11. 2025Principled Coarse-Grained Acceptance for Speculative Decoding in Speechcited by 2
  12. 2026MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesiscited by 1
5 more
  1. 2026DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generationcited by 1
  2. 2026WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Modelscited by 0
  3. 2026DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Executioncited by 0
  4. 2026TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speechcited by 0
  5. 2026Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding
  2. 2026Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
  3. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
  4. 2026Faster IndexTTS-2: Accelerating and Streaming Autoregressive Zero-Shot Text-to-Speech Synthesis on GPUs
  5. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
  6. 2026Acoustic Progress Propagation for Long-Horizon Speculative Decoding in ASR
  7. 2025Draft, Verify, and Improve: Toward Training-Aware Speculative Decoding
  8. 2026LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling