mlsanityData to 7 Oct 2026

Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems

arXiv 2407.03645 ↗First posted 4 Jul 2024cs.CL cs.SD eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

Continual Learning (CL) involves fine-tuning pre-trained models with new data while maintaining the performance on the pre-trained data. This is particularly relevant for expanding multilingual ASR (MASR) capabilities. However, existing CL methods, mainly designed for computer vision and reinforcement learning tasks, often yield sub-optimal results when directly applied to MASR. We hypothesise that this is because CL of the auto-regressive decoder in the MASR model is difficult. To verify this, we propose four optimizations on the decoder. They include decoder-layer gradient surgery, freezing unused token embeddings, suppressing output of newly added tokens, and learning rate re-scaling. Our experiments on adapting Whisper to 10 unseen languages from the Common Voice dataset demonstrate that these optimizations reduce the Average Word Error Rate (AWER) of pretrained languages from 14.2% to 12.4% compared with Experience Replay, without compromising the AWER of new languages.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

28 of 33 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2016Learning without Forgettingcited by 2,465
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  10. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  11. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  12. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
16 more
  1. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  2. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  3. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  5. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  6. 2020Stolen Probability: A Structural Weakness of Neural Language Modelscited by 21
  7. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  8. 2020Language-agnostic Multilingual Modelingcited by 17
  9. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 16
  10. 2022Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognitioncited by 10
  11. 2023Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learningcited by 9
  12. 2023Online Continual Learning in Keyword Spotting for Low-Resource Devices via Pooling High-Order Temporal Statisticscited by 9
  13. 2023Rehearsal-Free Online Continual Learning for Automatic Speech Recognitioncited by 7
  14. 2023Improving Replay Sample Selection and Storage for Less Forgetting in Continual Learningcited by 7
  15. 2022An Investigation of the Combination of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language Understandingcited by 5
  16. 2023Emphasized Non-Target Speaker Knowledge in Knowledge Distillation for Automatic Speaker Verificationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Efficient Multilingual ASR Finetuning via LoRA Language Expertscited by 5
  2. 2025Bona fide Cross Testing Reveals Weak Spot in Audio Deepfake Detection Systemscited by 3
  3. 2025Improving Synthetic Data Training for Contextual Biasing Models with a Keyword-Aware Cost Functioncited by 1
  4. 2026Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languagescited by 1
  5. 2025Efficient Trie-based Biasing using K-step Prediction for Rare Word Recognitioncited by 0
  6. 2026BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognitioncited by 0
  7. 2026A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speakscited by 0