mlsanityData to 14 Oct 2026

STAB: Speech Tokenizer Assessment Benchmark

arXiv 2409.02384 ↗First posted 4 Sep 2024cs.CL cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Representing speech as discrete tokens provides a framework for transforming speech into a format that closely resembles text, thus enabling the use of speech as an input to the widely successful large language models (LLMs). Currently, while several speech tokenizers have been proposed, there is ambiguity regarding the properties that are desired from a tokenizer for specific downstream tasks and its overall generalizability. Evaluating the performance of tokenizers across different downstream tasks is a computationally intensive effort that poses challenges for scalability. To circumvent this requirement, we present STAB (Speech Tokenizer Assessment Benchmark), a systematic evaluation framework designed to assess speech tokenizers comprehensively and shed light on their inherent characteristics. This framework provides a deeper understanding of the underlying mechanisms of speech tokenization, thereby offering a valuable resource for expediting the advancement of future tokenizer models and enabling comparative analysis using a standardized benchmark. We evaluate the STAB metrics and correlate this with downstream task performance across a range of speech tasks and tokenizer choices.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

30 of 44 references.

References · 30

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  7. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  10. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  12. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
18 more
  1. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  3. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  4. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  5. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  6. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  7. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  8. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 93
  9. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  10. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  11. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  12. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 71
  13. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  14. 2019Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networkscited by 20
  15. 2023Generative Spoken Language Model based on continuous word-sized audio tokenscited by 19
  16. 2020The Effect of Spectrogram Reconstruction on Automatic Music Transcription: An Alternative Approach to Improve Transcription Accuracycited by 10
  17. 2022Are word boundaries useful for unsupervised language learning?cited by 8
  18. 2015Alternative structures for character-level RNNscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  2. 2025Benchmarking Prosody Encoding in Discrete Speech Tokenscited by 1
  3. 2026Voxtral TTScited by 0
  4. 2026Content is What Remains: Invariant Speech Tokenization from Parallel Utterancescited by 0