mlsanityData to 14 Oct 2026

Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study

arXiv 2309.15800 ↗First posted 27 Sep 2023cs.CL cs.SD eess.AScited by 42

Not on the map. Why?

Abstract (arXiv)

Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies, evoking inefficiencies in sequence modeling. High-dimensional speech features such as spectrograms are often used as the input for the subsequent model. However, they can still be redundant. Recent investigations proposed the use of discrete speech units derived from self-supervised learning representations, which significantly compresses the size of speech data. Applying various methods, such as de-duplication and subword modeling, can further compress the speech sequence length. Hence, training time is significantly reduced while retaining notable performance. In this study, we undertake a comprehensive and systematic exploration into the application of discrete units within end-to-end speech processing models. Experiments on 12 automatic speech recognition, 3 speech translation, and 1 spoken language understanding corpora demonstrate that discrete units achieve reasonably good results in almost all the settings. We intend to release our configurations and trained models to foster future research efforts.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

35 of 46 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  11. 2022High Fidelity Neural Audio Compressioncited by 687
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
23 more
  1. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  2. 2015Attention-Based Models for Speech Recognitioncited by 401
  3. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  4. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  5. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  6. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  7. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  8. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  9. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  10. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  11. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  12. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  13. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  14. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 78
  15. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 71
  16. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  17. 2023Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learningcited by 28
  18. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  19. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 21
  20. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  21. 2021Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problemcited by 11
  22. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  23. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 19; 2025: 15; 2026: 6.

42 papers.

Sort:

By area:

  1. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  2. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 66
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  4. 2024Scaling Properties of Speech Language Modelscited by 32
  5. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  6. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
  7. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  8. 2025Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessmentcited by 15
  9. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  10. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  11. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  12. 2024LAST: Language Model Aware Speech Tokenizationcited by 9
30 more
  1. 2024Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representationscited by 8
  2. 2025From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLMcited by 8
  3. 2026Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspacescited by 8
  4. 2023TEAL: Tokenize and Embed ALL for Multi-modal Large Language Modelscited by 7
  5. 2024Exploring SSL Discrete Tokens for Multilingual ASRcited by 6
  6. 2025SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generationcited by 6
  7. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognitioncited by 5
  8. 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
  9. 2024Children's Speech Recognition through Discrete Token Enhancementcited by 4
  10. 2025Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Informationcited by 4
  11. 2025On-device Streaming Discrete Speech Unitscited by 4
  12. 2025Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMscited by 4
  13. 2024SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Datacited by 3
  14. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3
  15. 2025Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecscited by 3
  16. 2024Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selectioncited by 2
  17. 2024Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASRcited by 2
  18. 2024Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?cited by 2
  19. 2024Direct Punjabi to English speech translation using discrete unitscited by 1
  20. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1
  21. 2025Factorized RVQ-GAN For Disentangled Speech Tokenizationcited by 1
  22. 2025Autoregressive Speech Enhancement via Acoustic Tokenscited by 1
  23. 2025MLLM-based Speech Recognition: When and How is Multimodality Beneficial?cited by 1
  24. 2025Benchmarking Prosody Encoding in Discrete Speech Tokenscited by 1
  25. 2025Towards Leveraging Sequential Structure in Animal Vocalizationscited by 1
  26. 2026Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Taskscited by 1
  27. 2026Recovering Performance in Speech Emotion Recognition from Discrete Tokens via Multi-Layer Fusion and Paralinguistic Feature Integrationcited by 0
  28. 2026Frontend Token Enhancement for Token-Based Speech Recognitioncited by 0
  29. 2026STACodec: Semantic Token Assignment for Balancing Acoustic Fidelity and Semantic Information in Audio Codecscited by 0
  30. 2026TokAN: Accent Normalization Using Self-Supervised Speech Tokenscited by 0