mlsanityData to 7 Oct 2026

TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech

arXiv 2606.09019 ↗First posted 8 Jun 2026cs.SD cs.AIcited by 0

On the map: Efficient LLM inference and serving → Audio and omni-modal token compression

Abstract (arXiv)

Codec-based autoregressive (AR) speech language models have achieved strong text-to-speech (TTS) quality by modeling speech as sequences of discrete audio tokens with large pretrained backbones. However, this token-level formulation creates a structural efficiency bottleneck: speech-token sequences are much longer than text sequences, requiring the AR backbone to perform causal computation at every token position and maintain a KV cache that grows with the sequence length. We introduce TLDR, a patch-based autoregressive framework that accelerates codec-based AR-TTS by shifting the causal modeling from token-level speech sequences to patch-level sequences. TLDR groups consecutive codec tokens into compact latent patches using a lightweight compressor, models the resulting shorter patch sequence with a frozen pretrained AR-TTS backbone adapted by LoRA, and reconstructs fine-grained speech tokens within each patch using a speaker-conditioned extractor. With a patch size of 4, TLDR achieves a 1.8x inference speedup over the baseline AR-TTS model and reduces global KV-cache memory by up to 75%. Experimental results indicate that patch-level global causal modeling can be a practical way to reduce the inference cost of pretrained codec-based AR-TTS systems without replacing the existing modules.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 43 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

40 of 43 references.

References · 40

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  9. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  10. 2022High Fidelity Neural Audio Compressioncited by 673
  11. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
28 more
  1. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  2. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  3. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  4. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  5. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  6. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  7. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  8. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  9. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  10. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  11. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  12. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
  13. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  14. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  15. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 137
  16. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  17. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  18. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  19. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 92
  20. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 85
  21. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 69
  22. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
  23. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  24. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 56
  25. 2025FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbotcited by 34
  26. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  27. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  28. 2025Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decodingcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis
  2. 2026DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
  3. 2024Efficient Autoregressive Audio Modeling via Next-Scale Prediction
  4. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  5. 2025SyncSpeech: Efficient and Low-Latency Text-to-Speech based on Temporal Masked Transformer
  6. 2026Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens
  7. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesis
  8. 2025Next Tokens Denoising for Speech Synthesis