# LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR

arXiv: https://arxiv.org/abs/2406.06619 · HTML: https://mlsanity.com/paper/2406.06619/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Model](https://mlsanity.com/paper/1712.01541/) (2017)
- [Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual Speech Recognition](https://mlsanity.com/paper/2012.01687/) (2020)
- [A Configurable Multilingual Model is All You Need to Recognize All Languages](https://mlsanity.com/paper/2107.05876/) (2021)
- [Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition](https://mlsanity.com/paper/2307.05956/) (2023)
- [CL-MASR: A Continual Learning Benchmark for Multilingual ASR](https://mlsanity.com/paper/2310.16931/) (2023)
- [Language Adaptive Cross-lingual Speech Representation Learning with Sparse Sharing Sub-networks](https://mlsanity.com/paper/2203.04583/) (2022)
- [Learning ASR pathways: A sparse multilingual ASR model](https://mlsanity.com/paper/2209.05735/) (2022)
- [Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Training](https://mlsanity.com/paper/2303.00786/) (2023)
- [Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model](https://mlsanity.com/paper/2309.13018/) (2023)

## Built on by (most cited first, 30 of 36)

- [Low-Rank Adaptation for Foundation Models: A Comprehensive Review](https://mlsanity.com/paper/2501.00365/) (2024)
- [VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities](https://mlsanity.com/paper/2412.18161/) (2024)
- [Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding](https://mlsanity.com/paper/2412.16507/) (2024)
- [Efficient Multilingual ASR Finetuning via LoRA Language Experts](https://mlsanity.com/paper/2506.21555/) (2025)
- [Mixture of LoRA Experts for Low-Resourced Multi-Accent Automatic Speech Recognition](https://mlsanity.com/paper/2505.20006/) (2025)
- [Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning](https://mlsanity.com/paper/2506.21576/) (2025)
- [SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition](https://mlsanity.com/paper/2409.10429/) (2024)
- [Speech Recognition for Automatically Assessing Afrikaans and isiXhosa Preschool Oral Narratives](https://mlsanity.com/paper/2501.06478/) (2025)
- [SSVD: Structured SVD for Parameter-Efficient Fine-Tuning and Benchmarking under Domain Shift in ASR](https://mlsanity.com/paper/2509.02830/) (2025)
- [ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark](https://mlsanity.com/paper/2602.12911/) (2026)
- [BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASR](https://mlsanity.com/paper/2501.12602/) (2025)
- [Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR](https://mlsanity.com/paper/2506.21577/) (2025)
- [On the Role of Encoder Depth: Pruning Whisper and LoRA Fine-Tuning in SLAM-ASR](https://mlsanity.com/paper/2603.27981/) (2026)
- [A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenario](https://mlsanity.com/paper/2412.00721/) (2024)
- [SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challenge](https://mlsanity.com/paper/2507.03343/) (2025)
- [ILT-Iterative LoRA Training through Focus-Feedback-Fix for Multilingual Speech Recognition](https://mlsanity.com/paper/2507.08477/) (2025)
- [A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASR](https://mlsanity.com/paper/2601.00557/) (2026)
- [Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages](https://mlsanity.com/paper/2602.01008/) (2026)
- [Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognition](https://mlsanity.com/paper/2603.17558/) (2026)
- [Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)](https://mlsanity.com/paper/2606.07608/) (2026)
- [Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR](https://mlsanity.com/paper/2609.15758/) (2026)
- [Evaluating Automatic Speech Recognition Systems for Korean Meteorological Experts](https://mlsanity.com/paper/2410.18444/) (2024)
- [The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASR](https://mlsanity.com/paper/2510.22492/) (2025)
- [Parameter-Efficient Fine-Tuning for HAR: Integrating LoRA and QLoRA into Transformer Models](https://mlsanity.com/paper/2512.17983/) (2025)
- [Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition](https://mlsanity.com/paper/2606.06985/) (2026)
- [Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR](https://mlsanity.com/paper/2606.14391/) (2026)
- [Evaluating the Effect of Linguistic Relatedness on Cross-Lingual Transfer in Large Multilingual Automatic Speech Recognition](https://mlsanity.com/paper/2607.04814/) (2026)
- [Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders](https://mlsanity.com/paper/2607.07294/) (2026)
- [Robust Multi-Tier Infant-Centered Audio Understanding with Whisper via Structured Speaker Conditioning](https://mlsanity.com/paper/2608.11587/) (2026)
- [Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages](https://mlsanity.com/paper/2608.29239/) (2026)

## Abstract (arXiv)

Recent years have witnessed significant progress in multilingual automatic speech recognition (ASR), driven by the emergence of end-to-end (E2E) models and the scaling of multilingual datasets. Despite that, two main challenges persist in multilingual ASR: language interference and the incorporation of new languages without degrading the performance of the existing ones. This paper proposes LoRA-Whisper, which incorporates LoRA matrix into Whisper for multilingual ASR, effectively mitigating language interference. Furthermore, by leveraging LoRA and the similarities between languages, we can achieve better performance on new languages while upholding consistent performance on original ones. Experiments on a real-world task across eight languages demonstrate that our proposed LoRA-Whisper yields a relative gain of 18.5% and 23.0% over the baseline system for multilingual ASR and language expansion respectively.