mlsanityData to 7 Oct 2026

Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts

arXiv 2311.01070 ↗First posted 2 Nov 2023cs.CL cs.SD eess.AScited by 15

Not on the map. Why?

Abstract (arXiv)

Whisper is a multitask and multilingual speech model covering 99 languages. It yields commendable automatic speech recognition (ASR) results in a subset of its covered languages, but the model still underperforms on a non-negligible number of under-represented languages, a problem exacerbated in smaller model versions. In this work, we propose DistilWhisper, an approach able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. This dual approach allows us to effectively boost ASR performance while keeping the robustness inherited from the multitask and multilingual pre-training. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters, boosting performance in the targeted languages for both in- and out-of-domain test sets, while introducing only a negligible parameter overhead at inference.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 28 of its 35 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

28 of 35 references.

References · 28

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  10. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  11. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  12. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
16 more
  1. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  2. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  3. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  4. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  5. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  6. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  7. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  8. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  9. 2022Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 31
  10. 2021Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speechcited by 30
  11. 2022ESB: A Benchmark For Multi-Domain End-to-End Speech Recognitioncited by 18
  12. 2023PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Modelscited by 17
  13. 2022SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languagescited by 16
  14. 2023NAVER LABS Europe's Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Trackcited by 9
  15. 2023A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Modelcited by 8
  16. 2023Language-Universal Adapter Learning with Knowledge Distillation for End-to-End Multilingual Speech Recognitioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 6; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024Fine-tuning Whisper on Low-Resource Languages for Real-World Applicationscited by 11
  2. 2025Efficient Multilingual ASR Finetuning via LoRA Language Expertscited by 5
  3. 2024To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillationcited by 4
  4. 2024uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimescited by 3
  5. 2024Efficient Compression of Multitask Multilingual Speech Modelscited by 2
  6. 2025BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASRcited by 2
  7. 2025Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Modelscited by 2
  8. 2024From Random to Informed Data Selection: A Diversity-Based Approach to Optimize Human Annotation and Few-Shot Learningcited by 1
  9. 2024Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whispercited by 1
  10. 2025Advancing STT for Low-Resource Real-World Speechcited by 1
  11. 2025SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challengecited by 1
  12. 2026A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASRcited by 1
3 more
  1. 2024Evaluating Automatic Speech Recognition Systems for Korean Meteorological Expertscited by 0
  2. 2025Novel Parasitic Dual-Scale Modeling for Efficient and Accurate Multilingual Speech Translationcited by 0
  3. 2026Stride-k Subsampling: Train-Free Audio Token Reduction for Whispercited by 0