# Learn and Don't Forget: Adding a New Language to ASR Foundation Models

arXiv: https://arxiv.org/abs/2407.06800 · HTML: https://mlsanity.com/paper/2407.06800/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Scaling End-to-End Models for Large-Scale Multilingual ASR](https://mlsanity.com/paper/2104.14830/) (2021)
- [Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting](https://mlsanity.com/paper/1812.02464/) (2018)
- [Improving Massively Multilingual ASR With Auxiliary CTC Objectives](https://mlsanity.com/paper/2302.12829/) (2023)
- [Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation](https://mlsanity.com/paper/2305.10951/) (2023)
- [Adapting an ASR Foundation Model for Spoken Language Assessment](https://mlsanity.com/paper/2307.09378/) (2023)
- [Efficient Weight factorization for Multilingual Speech Recognition](https://mlsanity.com/paper/2105.03010/) (2021)
- [Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASR](https://mlsanity.com/paper/2110.04484/) (2021)
- [UML: A Universal Monolingual Output Layer for Multilingual ASR](https://mlsanity.com/paper/2302.11186/) (2023)
- [Towards continually learning new languages](https://mlsanity.com/paper/2211.11703/) (2022)
- [MixRep: Hidden Representation Mixup for Low-Resource Speech Recognition](https://mlsanity.com/paper/2310.18450/) (2023)

## Built on by (most cited first, 16 of 16)

- [Fine-tuning Whisper on Low-Resource Languages for Real-World Applications](https://mlsanity.com/paper/2412.15726/) (2024)
- [Task Arithmetic for Language Expansion in Speech Translation](https://mlsanity.com/paper/2409.11274/) (2024)
- [Weight Factorization and Centralization for Continual Learning in Speech Recognition](https://mlsanity.com/paper/2506.16574/) (2025)
- [Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning](https://mlsanity.com/paper/2506.21576/) (2025)
- [Scaling and Prompting for Improved End-to-End Spoken Grammatical Error Correction](https://mlsanity.com/paper/2505.21137/) (2025)
- [Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR](https://mlsanity.com/paper/2506.21577/) (2025)
- [Multimodal In-context Learning for ASR of Low-resource Languages](https://mlsanity.com/paper/2601.05707/) (2026)
- [Advancing STT for Low-Resource Real-World Speech](https://mlsanity.com/paper/2506.08836/) (2025)
- [End-to-End Spoken Grammatical Error Correction](https://mlsanity.com/paper/2506.18532/) (2025)
- [MLMA: Towards Multilingual ASR With Mamba-based Architectures](https://mlsanity.com/paper/2510.18684/) (2025)
- [Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?](https://mlsanity.com/paper/2608.12332/) (2026)
- [Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR](https://mlsanity.com/paper/2609.15758/) (2026)
- [Navigating the Reality Gap: On-Device Continual Adaptation of ASR for Clinical Telephony](https://mlsanity.com/paper/2512.16401/) (2025)
- [BlasBench: An Open Benchmark for Irish Speech Recognition](https://mlsanity.com/paper/2604.10736/) (2026)
- [Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASR](https://mlsanity.com/paper/2607.11163/) (2026)
- [Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognition](https://mlsanity.com/paper/2609.29800/) (2026)

## Abstract (arXiv)

Foundation ASR models often support many languages, e.g. 100 languages in Whisper. However, there has been limited work on integrating an additional, typically low-resource, language, while maintaining performance on the original language set. Fine-tuning, while simple, may degrade the accuracy of the original set. We compare three approaches that exploit adaptation parameters: soft language code tuning, train only the language code; soft prompt tuning, train prepended tokens; and LoRA where a small set of additional parameters are optimised. Elastic Weight Consolidation (EWC) offers an alternative compromise with the potential to maintain performance in specific target languages. Results show that direct fine-tuning yields the best performance for the new language but degrades existing language capabilities. EWC can address this issue for specific languages. If only adaptation parameters are used, the language capabilities are maintained but at the cost of performance in the new language.