mlsanityData to 7 Oct 2026

On the Effectiveness of Acoustic BPE in Decoder-Only TTS

arXiv 2407.03892 ↗First posted 4 Jul 2024cs.SD cs.AI eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Discretizing speech into tokens and generating them by a decoder-only model have been a promising direction for text-to-speech (TTS) and spoken language modeling (SLM). To shorten the sequence length of speech tokens, acoustic byte-pair encoding (BPE) has emerged in SLM that treats speech tokens from self-supervised semantic representations as characters to further compress the token sequence. But the gain in TTS has not been fully investigated, and the proper choice of acoustic BPE remains unclear. In this work, we conduct a comprehensive study on various settings of acoustic BPE to explore its effectiveness in decoder-only TTS models with semantic speech tokens. Experiments on LibriTTS verify that acoustic BPE uniformly increases the intelligibility and diversity of synthesized speech, while showing different features across BPE settings. Hence, acoustic BPE is a favorable tool for decoder-only TTS.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 27 of its 31 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

27 of 31 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  8. 2022High Fidelity Neural Audio Compressioncited by 673
  9. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  12. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
15 more
  1. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  2. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  3. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  4. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  5. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  6. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  7. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 77
  8. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  9. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  10. 2018On GANs and GMMscited by 62
  11. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  12. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  13. 2023Acoustic BPE for Speech Generation with Discrete Tokenscited by 15
  14. 2022Speech Pre-training with Acoustic Piececited by 14
  15. 2023Vec-Tok Speech: speech vectorization and tokenization for neural speech generationcited by 12

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  3. 2024LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codeccited by 7
  4. 2025Latent Speech-Text Transformercited by 4