mlsanityData to 7 Oct 2026

Exploring SSL Discrete Tokens for Multilingual ASR

arXiv 2409.08805 ↗First posted 13 Sep 2024cs.CL cs.SD eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

With the advancement of Self-supervised Learning (SSL) in speech-related tasks, there has been growing interest in utilizing discrete tokens generated by SSL for automatic speech recognition (ASR), as they offer faster processing techniques. However, previous studies primarily focused on multilingual ASR with Fbank features or English ASR with discrete tokens, leaving a gap in adapting discrete tokens for multilingual ASR scenarios. This study presents a comprehensive comparison of discrete tokens generated by various leading SSL models across multiple language domains. We aim to explore the performance and efficiency of speech discrete tokens across multiple language domains for both monolingual and multilingual ASR scenarios. Experimental results demonstrate that discrete tokens achieve comparable results against systems trained on Fbank features in ASR tasks across seven language domains with an average word error rate (WER) reduction of 0.31% and 1.76% absolute (2.80% and 15.70% relative) on dev and test sets respectively, with particularly WER reduction of 6.82% absolute (41.48% relative) on the Polish test set.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

29 of 35 references.

References · 29

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  9. 2022High Fidelity Neural Audio Compressioncited by 673
  10. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
17 more
  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  3. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  4. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  5. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  6. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 77
  7. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  8. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  9. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  10. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  11. 2019Transformer ASR with Contextual Block Processingcited by 28
  12. 2023Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learningcited by 28
  13. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  14. 2023Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTScited by 24
  15. 2024The Interspeech 2024 Challenge on Speech Processing Using Discrete Unitscited by 12
  16. 2023Towards Effective and Compact Contextual Representation for Conformer Transducer Speech Recognition Systemscited by 6
  17. 2022Confidence Score Based Conformer Speaker Adaptation for Speech Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2025Beyond WER: Probing Whisper's Sub-token Decoder Across Diverse Language Resource Levelscited by 2
  3. 2025The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASRcited by 0
  4. 2026Controllable Accent Normalization via Discrete Diffusioncited by 0
  5. 2026TokAN: Accent Normalization Using Self-Supervised Speech Tokenscited by 0
  6. 2026TokenMapper: A Step Toward Interoperable Speech Token Translationcited by 0