mlsanityData to 7 Oct 2026

WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models

arXiv 2604.08558 ↗First posted 17 Mar 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent decoder-only autoregressive text-to-speech (AR-TTS) models produce high-fidelity speech, but their memory and compute costs scale quadratically with sequence length due to full self-attention. In this paper, we propose WAND, Windowed Attention and Knowledge Distillation, a framework that adapts pretrained AR-TTS models to operate with constant computational and memory complexity. WAND separates the attention mechanism into two: persistent global attention over conditioning tokens and local sliding-window attention over generated tokens. To stabilize fine-tuning, we employ a curriculum learning strategy that progressively tightens the attention window. We further utilize knowledge distillation from a full-attention teacher to recover high-fidelity synthesis quality with high data efficiency. Evaluated on three modern AR-TTS models, WAND preserves the original quality while achieving up to 66.2% KV cache memory reduction and length-invariant, near-constant per-step latency.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 43 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

40 of 43 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  11. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  12. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
28 more
  1. 2022High Fidelity Neural Audio Compressioncited by 673
  2. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  3. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  4. 2022Efficiently Scaling Transformer Inferencecited by 529
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  6. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  7. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  8. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  9. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  10. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  11. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  12. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  13. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  14. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  15. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 92
  16. 2024CLLMs: Consistency Large Language Modelscited by 71
  17. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  18. 2025IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech Systemcited by 27
  19. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesiscited by 22
  20. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  21. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  22. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  23. 2021Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognitioncited by 11
  24. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  25. 2023Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speechcited by 3
  26. 2025SPADE: Structured Pruning and Adaptive Distillation for Efficient LLM-TTScited by 2
  27. 2024Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesiscited by 1
  28. 2026MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesiscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Nix-TTS: Lightweight and End-to-End Text-to-Speech via Module-wise Distillation
  2. 2026TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech
  3. 2025CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis
  4. 2026LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling
  5. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
  6. 2024When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
  7. 2026MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control
  8. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents