mlsanityData to 14 Oct 2026

Learn and Don't Forget: Adding a New Language to ASR Foundation Models

arXiv 2407.06800 ↗First posted 9 Jul 2024eess.AS cs.CL cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Foundation ASR models often support many languages, e.g. 100 languages in Whisper. However, there has been limited work on integrating an additional, typically low-resource, language, while maintaining performance on the original language set. Fine-tuning, while simple, may degrade the accuracy of the original set. We compare three approaches that exploit adaptation parameters: soft language code tuning, train only the language code; soft prompt tuning, train prepended tokens; and LoRA where a small set of additional parameters are optimised. Elastic Weight Consolidation (EWC) offers an alternative compromise with the potential to maintain performance in specific target languages. Results show that direct fine-tuning yields the best performance for the new language but degrades existing language capabilities. EWC can address this issue for specific languages. If only adaptation parameters are used, the language capabilities are maintained but at the cost of performance in the new language.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

29 of 45 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  11. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  12. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
17 more
  1. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  2. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  4. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  5. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  6. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  7. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  8. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  9. 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
  10. 2023Improving Massively Multilingual ASR With Auxiliary CTC Objectivescited by 31
  11. 2023Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentationcited by 31
  12. 2023Adapting an ASR Foundation Model for Spoken Language Assessmentcited by 14
  13. 2021Efficient Weight factorization for Multilingual Speech Recognitioncited by 12
  14. 2021Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASRcited by 7
  15. 2023UML: A Universal Monolingual Output Layer for Multilingual ASRcited by 3
  16. 2022Towards continually learning new languagescited by 2
  17. 2023MixRep: Hidden Representation Mixup for Low-Resource Speech Recognitioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 6.

16 papers.

Sort:

By area:

  1. 2024Fine-tuning Whisper on Low-Resource Languages for Real-World Applicationscited by 11
  2. 2024Task Arithmetic for Language Expansion in Speech Translationcited by 6
  3. 2025Weight Factorization and Centralization for Continual Learning in Speech Recognitioncited by 6
  4. 2025Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuningcited by 5
  5. 2025Scaling and Prompting for Improved End-to-End Spoken Grammatical Error Correctioncited by 2
  6. 2025Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASRcited by 2
  7. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 2
  8. 2025Advancing STT for Low-Resource Real-World Speechcited by 1
  9. 2025End-to-End Spoken Grammatical Error Correctioncited by 1
  10. 2025MLMA: Towards Multilingual ASR With Mamba-based Architecturescited by 1
  11. 2026Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?cited by 1
  12. 2026Sequential Adapter Stacking for Cross-Lingual Low-Resource ASRcited by 1
4 more
  1. 2025Navigating the Reality Gap: On-Device Continual Adaptation of ASR for Clinical Telephonycited by 0
  2. 2026BlasBench: An Open Benchmark for Irish Speech Recognitioncited by 0
  3. 2026Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASRcited by 0
  4. 2026Adaptive Fisher-Whitened Cross-Covariance for Low-Resource Speech Recognitioncited by 0