mlsanityData to 14 Oct 2026

Weighted Cross-entropy for Low-Resource Languages in Multilingual Speech Recognition

arXiv 2409.16954 ↗First posted 25 Sep 2024cs.CL cs.SD eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

This paper addresses the challenge of integrating low-resource languages into multilingual automatic speech recognition (ASR) systems. We introduce a novel application of weighted cross-entropy, typically used for unbalanced datasets, to facilitate the integration of low-resource languages into pre-trained multilingual ASR models within the context of continual multilingual learning. We fine-tune the Whisper multilingual ASR model on five high-resource languages and one low-resource language, employing language-weighted dynamic cross-entropy and data augmentation. The results show a remarkable 6.69% word error rate (WER) reduction for the low-resource language compared to the fine-tuned model without applying our approach, and a 48.86% WER reduction compared to the original Whisper model. In addition, our approach yields an average WER reduction of 3.29% across the six languages, showing no degradation for the high-resource languages.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 14 of its 25 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

14 of 25 references.

References · 14

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  5. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  6. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  7. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  8. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  9. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  10. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 73
  11. 2021MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognitioncited by 39
  12. 2015Transfer Learning for Speech and Language Processingcited by 35
2 more
  1. 2018Toward domain-invariant speech recognition via large scale trainingcited by 35
  2. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2024Fine-tuning Whisper on Low-Resource Languages for Real-World Applicationscited by 11
  2. 2025Efficient Multilingual ASR Finetuning via LoRA Language Expertscited by 6
  3. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 5
  4. 2025Advancing STT for Low-Resource Real-World Speechcited by 1
  5. 2026CORE-Acu: Structured Reasoning Traces and Knowledge Graph Safety Verification for Acupuncture Clinical Decision Supportcited by 0
  6. 2026When Surveys Become Conversations: Adaptive Matrix Validation for AI-Assisted Interviewscited by 0