mlsanityData to 7 Oct 2026

Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding

arXiv 2410.13839 ↗First posted 17 Oct 2024cs.SD cs.AI eess.AScited by 10

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

The goal of this paper is to accelerate codec-based speech synthesis systems with minimum sacrifice to speech quality. We propose an enhanced inference method that allows for flexible trade-offs between speed and quality during inference without requiring additional training. Our core idea is to predict multiple tokens per inference step of the AR module using multiple prediction heads, resulting in a linear reduction in synthesis time as the number of heads increases. Furthermore, we introduce a novel speculative decoding technique that utilises a Viterbi-based algorithm to select the optimal sequence of generated tokens at each decoding step. In our experiments, we demonstrate that the time required to predict each token is reduced by a factor of 4 to 5 compared to baseline models, with minimal quality trade-off or even improvement in terms of speech intelligibility. Audio samples are available at: multpletokensprediction.github.io/multipletokensprediction.github.io/.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 44 of its 48 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

44 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  2. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  3. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 41

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  5. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  6. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  7. 2022High Fidelity Neural Audio Compressioncited by 673
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  9. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  11. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  12. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
29 more
  1. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  3. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  4. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  5. 2023Listen, Think, and Understandcited by 206
  6. 2021A Survey on Neural Speech Synthesiscited by 182
  7. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  8. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  9. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  10. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  11. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  12. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  13. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  14. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  15. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  16. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 84
  17. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  18. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 70
  19. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  20. 2023CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Modelcited by 44
  21. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  22. 2024CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speechcited by 37
  23. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 37
  24. 2023HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayescited by 27
  25. 2021Fre-GAN: Adversarial Frequency-consistent Audio Synthesiscited by 24
  26. 2024VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speechcited by 23
  27. 2024UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learnercited by 22
  28. 2024FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocodercited by 11
  29. 2024dMel: Speech Tokenization made Simplecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decodingcited by 6
  2. 2025Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignmentcited by 4
  3. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  4. 2025CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidancecited by 3
  5. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  6. 2025Principled Coarse-Grained Acceptance for Speculative Decoding in Speechcited by 2
  7. 2026MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesiscited by 1
  8. 2026WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Modelscited by 0
  9. 2026TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speechcited by 0
  10. 2026Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding
  2. 2026One-Step Token-to-Waveform Generation with MeanFlow in Latent Space ↗
  3. 2025Model-free Speculative Decoding for Transformer-based ASR with Token Map Drafting
  4. 2026DLLM-TTS: Block Discrete Diffusion Language Model for Text-to-Speech Synthesis
  5. 2026X-Pred MeanFlow for Streaming Token-to-Mel Speech Decoding ↗
  6. 2026An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation
  7. 2026Acoustic Progress Propagation for Long-Horizon Speculative Decoding in ASR
  8. 2024A Theoretical Perspective for Speculative Decoding Algorithm