mlsanityData to 14 Oct 2026

Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding

arXiv 2505.15380 ↗First posted 21 May 2025cs.SD cs.AI eess.AScited by 6

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Modern autoregressive speech synthesis models leveraging language models have demonstrated remarkable performance. However, the sequential nature of next token prediction in these models leads to significant latency, hindering their deployment in scenarios where inference speed is critical. In this work, we propose Speech Speculative Decoding (SSD), a novel framework for autoregressive speech synthesis acceleration. Specifically, our method employs a lightweight draft model to generate candidate token sequences, which are subsequently verified in parallel by the target model using the proposed SSD framework. Experimental results demonstrate that SSD achieves a significant speedup of 1.4x compared with conventional autoregressive decoding, while maintaining high fidelity and naturalness. Subjective evaluations further validate the effectiveness of SSD in preserving the perceptual quality of the target model while accelerating inference.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 31 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

31 of 31 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding29 Oct
  2. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding17 Oct
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  4. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  5. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  7. 2022High Fidelity Neural Audio Compressioncited by 687
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  9. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  10. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  11. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  12. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
14 more
  1. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  4. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  5. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  6. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  7. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  8. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  9. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  10. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
  11. 2024Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesiscited by 48
  12. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  13. 2022Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 32
  14. 2023VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Designcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  2. 2025Principled Coarse-Grained Acceptance for Speculative Decoding in Speechcited by 2
  3. 2026VocalNet-MDM: Accelerating Streaming Speech LLM via Self-Distilled Masked Diffusion Modelingcited by 1
  4. 2026SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Modelscited by 0
  5. 2026TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speechcited by 0
  6. 2026Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniques
  2. 2026Acoustic Progress Propagation for Long-Horizon Speculative Decoding in ASR
  3. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  4. 2025Confidence-Modulated Speculative Decoding for Large Language Models
  5. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
  6. 2026Faster IndexTTS-2: Accelerating and Streaming Autoregressive Zero-Shot Text-to-Speech Synthesis on GPUs
  7. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation
  8. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding