mlsanityData to 14 Oct 2026

Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models

arXiv 2505.17446 ↗First posted 23 May 2025cs.CL cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

The purpose of speech tokenization is to transform a speech signal into a sequence of discrete representations, serving as the foundation for speech language models (SLMs). While speech tokenization has many options, their effect on the performance of SLMs remains unclear. This paper investigates two key aspects of speech tokenization: the segmentation width and the cluster size of discrete units. First, we segment speech signals into fixed/variable widths and pooled representations. We then train K-means models in multiple cluster sizes. Through the evaluation on zero-shot spoken language understanding benchmarks, we find the positive effect of moderately coarse segmentation and bigger cluster size. Notably, among the best-performing models, the most efficient one achieves a 50% reduction in training data and a 70% decrease in training runtime. Our analysis highlights the importance of combining multiple tokens to enhance fine-grained spoken language understanding.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 25 of its 28 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

25 of 28 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  8. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  9. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  10. 2021Direct speech-to-speech translation with discrete unitscited by 149
  11. 2022Generative Spoken Dialogue Language Modelingcited by 126
  12. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
13 more
  1. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  2. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  3. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  4. 2023Textually Pretrained Speech Language Modelscited by 101
  5. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  6. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
  7. 2023ProsAudit, a prosodic benchmark for self-supervised speech modelscited by 25
  8. 2022Are discrete units necessary for Spoken Language Modeling?cited by 24
  9. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  10. 2023Generative Spoken Language Model based on continuous word-sized audio tokenscited by 19
  11. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  12. 2023Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Predictioncited by 17
  13. 2023Unsupervised Word Segmentation Using Temporal Gradient Pseudo-Labelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Benchmarking Prosody Encoding in Discrete Speech Tokenscited by 1
  2. 2026Segment Length Matters: A Study of Segment Lengths on Audio Fingerprinting Performancecited by 1
  3. 2026On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Modelscited by 0
  4. 2026Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenizationcited by 0