# A Parameter-efficient Language Extension Framework for Multilingual ASR

arXiv: https://arxiv.org/abs/2406.06329 · HTML: https://mlsanity.com/paper/2406.06329/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Progressive Prompts: Continual Learning for Language Models](https://mlsanity.com/paper/2301.12314/) (2023)
- [Continual Lifelong Learning in Natural Language Processing: A Survey](https://mlsanity.com/paper/2012.09823/) (2020)
- [Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters](https://mlsanity.com/paper/2007.03001/) (2020)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [CL-MASR: A Continual Learning Benchmark for Multilingual ASR](https://mlsanity.com/paper/2310.16931/) (2023)
- [Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition](https://mlsanity.com/paper/2210.15282/) (2022)
- [Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learning](https://mlsanity.com/paper/2306.15686/) (2023)
- [Continual Learning for Monolingual End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2112.09427/) (2021)
- [Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing](https://mlsanity.com/paper/2211.01522/) (2022)

## Built on by (most cited first, 3 of 3)

- [Efficient Multilingual ASR Finetuning via LoRA Language Experts](https://mlsanity.com/paper/2506.21555/) (2025)
- [BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASR](https://mlsanity.com/paper/2501.12602/) (2025)
- [On the Difficulty of Token-Level Modeling of Dysfluency and Fluency Shaping Artifacts](https://mlsanity.com/paper/2512.02027/) (2025)

## Abstract (arXiv)

Covering all languages with a multilingual speech recognition model (MASR) is very difficult. Performing language extension on top of an existing MASR is a desirable choice. In this study, the MASR continual learning problem is probabilistically decomposed into language identity prediction (LP) and cross-lingual adaptation (XLA) sub-problems. Based on this, we propose an architecture-based framework for language extension that can fundamentally solve catastrophic forgetting, debudded as PELE. PELE is designed to be parameter-efficient, incrementally incorporating an add-on module to adapt to a new language. Specifically, different parameter-efficient fine-tuning (PEFT) modules and their variants are explored as potential candidates to perform XLA. Experiments are carried out on 5 new languages with a wide range of low-resourced data sizes. The best-performing PEFT candidate can achieve satisfactory performance across all languages and demonstrates superiority in three of five languages over the continual joint learning setting. Notably, PEFT methods focusing on weight parameters or input features are revealed to be limited in performance, showing significantly inferior extension capabilities compared to inserting a lightweight module in between layers such as an Adapter.