mlsanityData to 7 Oct 2026

TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizers

arXiv 2406.15752 ↗First posted 22 Jun 2024eess.AS cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Neural codec language model (LM) has demonstrated strong capability in zero-shot text-to-speech (TTS) synthesis. However, the codec LM often suffers from limitations in inference speed and stability, due to its auto-regressive nature and implicit alignment between text and audio. In this work, to handle these challenges, we introduce a new variant of neural codec LM, namely TacoLM. Specifically, TacoLM introduces a gated attention mechanism to improve the training and inference efficiency and reduce the model size. Meanwhile, an additional gated cross-attention layer is included for each decoder layer, which improves the efficiency and content accuracy of the synthesized speech. In the evaluation of the Librispeech corpus, the proposed TacoLM achieves a better word error rate, speaker similarity, and mean opinion score, with 90% fewer parameters and 5.2 times speed up, compared with VALL-E. Demo and code is available at https://ereboas.github.io/TacoLM/.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 27 references that are in the map. First posted on arXiv 22 Jun 2024.

Builds on

23 of 27 references.

References · 23

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  10. 2022High Fidelity Neural Audio Compressioncited by 673
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
11 more
  1. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  2. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  3. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  4. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  5. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  6. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  7. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  8. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  9. 2021SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Modelcited by 56
  10. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  11. 2023Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decodercited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2025Voice Cloning: Comprehensive Surveycited by 4
  3. 2025Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Ratecited by 4
  4. 2026V.O.I.C.E (Voice, Ownership, Identity, Control, Expression): Risk Taxonomy of Synthetic Voice Generation From Empirical Datacited by 0