mlsanityData to 14 Oct 2026

Exploring the Benefits of Tokenization of Discrete Acoustic Units

arXiv 2406.05547 ↗First posted 8 Jun 2024cs.SD cs.CL eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Tokenization algorithms that merge the units of a base vocabulary into larger, variable-rate units have become standard in natural language processing tasks. This idea, however, has been mostly overlooked when the vocabulary consists of phonemes or Discrete Acoustic Units (DAUs), an audio-based representation that is playing an increasingly important role due to the success of discrete language-modeling techniques. In this paper, we showcase the advantages of tokenization of phonetic units and of DAUs on three prediction tasks: grapheme-to-phoneme, grapheme-to-DAUs, and unsupervised speech generation using DAU language modeling. We demonstrate that tokenization yields significant improvements in terms of performance, as well as training and inference speed, across all three tasks. We also offer theoretical insights to provide some explanation for the superior performance observed.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 8 Jun 2024.

Builds on

39 of 50 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2024Mixtral of Expertscited by 1,823
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
27 more
  1. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  2. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  4. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  5. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  6. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  7. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  8. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  9. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  10. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  11. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  12. 2021Direct speech-to-speech translation with discrete unitscited by 149
  13. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  14. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  15. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 101
  16. 2023Textually Pretrained Speech Language Modelscited by 101
  17. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  18. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  19. 2022ByT5 model for massively multilingual grapheme-to-phoneme conversioncited by 49
  20. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  21. 2018Improved training of end-to-end attention models for speech recognitioncited by 36
  22. 2021Variable-rate discrete representation learningcited by 24
  23. 2022SoundChoice: Grapheme-to-Phoneme Models with Semantic Disambiguationcited by 14
  24. 2021Acoustic Data-Driven Subword Modeling for End-to-End Speech Recognitioncited by 11
  25. 2023Speak While You Think: Streaming Speech Synthesis During Text Generationcited by 8
  26. 2024Data-driven grapheme-to-phoneme representations for a lexicon-free text-to-speechcited by 3
  27. 2024Spiking Music: Audio Compression with Event Based Auto-encoderscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  2. 2025Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMscited by 4
  3. 2024A Comparative Study of Discrete Speech Tokens for Semantic-Related Tasks with Large Language Modelscited by 3
  4. 2026On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Modelscited by 0