Improving Whisper's Recognition Performance for Under-Represented Language Kazakh Leveraging Unpaired Speech and Text
Not on the map. Why?
Abstract (arXiv)
Whisper and other large-scale automatic speech recognition models have made significant progress in performance. However, their performance on many low-resource languages, such as Kazakh, is not satisfactory. It is worth researching how to utilize low-cost data to improve the performance of Whisper on under-represented languages. In this study, we utilized easily accessible unpaired speech and text data and combined the language model GPT with Whisper on Kazakh. We implemented end of transcript (EOT) judgment modification and hallucination penalty to improve the performance of speech recognition. Further, we employed the decoding average token log probability as a criterion to select samples from unlabeled speech data and used pseudo-labeled data to fine-tune the model to further improve its performance. Ultimately, we achieved more than 10% absolute WER reduction in multiple experiments, and the whole process has the potential to be generalized to other under-represented languages.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 19 of its 23 references that are in the map. First posted on arXiv 10 Aug 2024.
Builds on
19 of 23 references.
References · 19
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
- 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
- 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
- 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
- 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
- 2021Unsupervised Speech Recognitioncited by 147
- 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
- 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
7 more
- 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
- 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 81
- 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
- 2022Towards End-to-end Unsupervised Speech Recognitioncited by 44
- 2017Transfer Learning for Speech Recognition on a Budgetcited by 25
- 2023Can Contextual Biasing Remain Effective with Whisper and GPT-2?cited by 21
- 2018Sequence-based Multi-lingual Low Resource Speech Recognitioncited by 20
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.