mlsanityData to 14 Oct 2026

LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR

arXiv 2406.06619 ↗First posted 7 Jun 2024eess.AS cs.AI cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed significant progress in multilingual automatic speech recognition (ASR), driven by the emergence of end-to-end (E2E) models and the scaling of multilingual datasets. Despite that, two main challenges persist in multilingual ASR: language interference and the incorporation of new languages without degrading the performance of the existing ones. This paper proposes LoRA-Whisper, which incorporates LoRA matrix into Whisper for multilingual ASR, effectively mitigating language interference. Furthermore, by leveraging LoRA and the similarities between languages, we can achieve better performance on new languages while upholding consistent performance on original ones. Experiments on a real-world task across eight languages demonstrate that our proposed LoRA-Whisper yields a relative gain of 18.5% and 23.0% over the baseline system for multilingual ASR and language expansion respectively.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

27 of 37 references.

References · 27

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  6. 2018Experience Replay for Continual Learningcited by 1,095
  7. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  8. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  9. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  10. 2015Listen, Attend and Spellcited by 543
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  12. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
15 more
  1. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  2. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  3. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  4. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  5. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  6. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 93
  7. 2017Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Modelcited by 37
  8. 2020Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual Speech Recognitioncited by 35
  9. 2021A Configurable Multilingual Model is All You Need to Recognize All Languagescited by 28
  10. 2023Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognitioncited by 26
  11. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 17
  12. 2022Language Adaptive Cross-lingual Speech Representation Learning with Sparse Sharing Sub-networkscited by 13
  13. 2022Learning ASR pathways: A sparse multilingual ASR modelcited by 12
  14. 2023Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Trainingcited by 4
  15. 2023Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Modelcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 11; 2026: 19.

36 papers.

Sort:

By area:

  1. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  2. 2024VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilitiescited by 10
  3. 2024Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decodingcited by 7
  4. 2025Efficient Multilingual ASR Finetuning via LoRA Language Expertscited by 6
  5. 2025Mixture of LoRA Experts for Low-Resourced Multi-Accent Automatic Speech Recognitioncited by 5
  6. 2025Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuningcited by 5
  7. 2024SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognitioncited by 4
  8. 2025Speech Recognition for Automatically Assessing Afrikaans and isiXhosa Preschool Oral Narrativescited by 4
  9. 2025SSVD: Structured SVD for Parameter-Efficient Fine-Tuning and Benchmarking under Domain Shift in ASRcited by 4
  10. 2026ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmarkcited by 3
  11. 2025BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASRcited by 2
  12. 2025Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASRcited by 2
24 more
  1. 2026On the Role of Encoder Depth: Pruning Whisper and LoRA Fine-Tuning in SLAM-ASRcited by 2
  2. 2024A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenariocited by 1
  3. 2025SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challengecited by 1
  4. 2025ILT-Iterative LoRA Training through Focus-Feedback-Fix for Multilingual Speech Recognitioncited by 1
  5. 2026A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASRcited by 1
  6. 2026Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languagescited by 1
  7. 2026Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognitioncited by 1
  8. 2026Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)cited by 1
  9. 2026Sequential Adapter Stacking for Cross-Lingual Low-Resource ASRcited by 1
  10. 2024Evaluating Automatic Speech Recognition Systems for Korean Meteorological Expertscited by 0
  11. 2025The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASRcited by 0
  12. 2025Parameter-Efficient Fine-Tuning for HAR: Integrating LoRA and QLoRA into Transformer Modelscited by 0
  13. 2026Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognitioncited by 0
  14. 2026Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASRcited by 0
  15. 2026Evaluating the Effect of Linguistic Relatedness on Cross-Lingual Transfer in Large Multilingual Automatic Speech Recognitioncited by 0
  16. 2026Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoderscited by 0
  17. 2026Robust Multi-Tier Infant-Centered Audio Understanding with Whisper via Structured Speaker Conditioningcited by 0
  18. 2026Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languagescited by 0
  19. 2026Automatic Lyric Transcription for Greek Songs: Scaling and Task Composition Effects in Whisper Adaptationcited by 0
  20. 2026Whisper-Based Speech Transcription from Videos Across Multiple Languages for Cross-Cultural Understandingcited by 0
  21. 2026DiaWhisper-DPO: Role-Attributed Transcription of Clinical Interviews via Failure-Mined Preference Optimizationcited by 0
  22. 2026T-SANDHI: Tone Sandhi-aware Adaptive Network with Decoupled Hybrid Injection for Low-resource Taiwanese Hokkien Speech Recognitioncited by 0
  23. 2026Beyond WER: Entity and Disfluency Recall in Accented Conversational ASRcited by 0
  24. 2026Understanding Personalization Gains in Whispercited by 0