mlsanityData to 7 Oct 2026

Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition

arXiv 2210.15282 ↗First posted 27 Oct 2022eess.AS cs.CL cs.SDcited by 10

Not on the map. Why?

Abstract (arXiv)

Adapting a trained Automatic Speech Recognition (ASR) model to new tasks results in catastrophic forgetting of old tasks, limiting the model's ability to learn continually and to be extended to new speakers, dialects, languages, etc. Focusing on End-to-End ASR, in this paper, we propose a simple yet effective method to overcome catastrophic forgetting: weight averaging. By simply taking the average of the previous and the adapted model, our method achieves high performance on both the old and new tasks. It can be further improved by introducing a knowledge distillation loss during the adaptation. We illustrate the effectiveness of our method on both monolingual and multilingual ASR. In both cases, our method strongly outperforms all baselines, even in its simplest form.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 28 of its 31 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

28 of 31 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2016Learning without Forgettingcited by 2,465
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  12. 2016Progressive Neural Networkscited by 1,332
16 more
  1. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  2. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  3. 2018Experience Replay for Continual Learningcited by 1,074
  4. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  7. 2020What is being transferred in transfer learning?cited by 365
  8. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  9. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  10. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  11. 2021Towards Lifelong Learning of End-to-end ASRcited by 20
  12. 2022Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognitioncited by 15
  13. 2022Online Continual Learning of End-to-End Speech Recognition Modelscited by 15
  14. 2021Continual Learning for Monolingual End-to-End Automatic Speech Recognitioncited by 8
  15. 2016The AMU-UEDIN Submission to the WMT16 News Translation Task: Attention-based NMT Models as Feature Functions in Phrase-based SMTcited by 6
  16. 2019Cued@wmt19:ewc&lmscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 6.

10 papers.

  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  3. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  4. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  5. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  6. 2024Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systemscited by 7
  7. 2024A Parameter-efficient Language Extension Framework for Multilingual ASRcited by 3
  8. 2023The Gift of Feedback: Improving ASR Model Quality by Learning from User Corrections through Federated Learningcited by 2
  9. 2024Sequential Editing for Lifelong Training of Speech Recognition Modelscited by 1
  10. 2023SRIL: Selective Regularization for Class-Incremental Learningcited by 0