mlsanityData to 7 Oct 2026

Online Continual Learning of End-to-End Speech Recognition Models

arXiv 2207.05071 ↗First posted 11 Jul 2022cs.LG cs.AI cs.SDcited by 15

Not on the map. Why?

Abstract (arXiv)

Continual Learning, also known as Lifelong Learning, aims to continually learn from new data as it becomes available. While prior research on continual learning in automatic speech recognition has focused on the adaptation of models across multiple different speech recognition tasks, in this paper we propose an experimental setting for textitonline continual learning for automatic speech recognition of a single task. Specifically focusing on the case where additional training data for the same task becomes available incrementally over time, we demonstrate the effectiveness of performing incremental model updates to end-to-end speech recognition models with an online Gradient Episodic Memory (GEM) method. Moreover, we show that with online continual learning and a selective sampling strategy, we can maintain an accuracy that is similar to retraining a model from scratch while requiring significantly lower computation costs. We have also verified our method with self-supervised learning (SSL) features.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 11 Jul 2022.

Builds on

23 of 38 references.

References · 23

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2016Learning without Forgettingcited by 2,465
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  5. 2016Progressive Neural Networkscited by 1,332
  6. 2017Continual Learning with Deep Generative Replaycited by 1,152
  7. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  8. 2018Experience Replay for Continual Learningcited by 1,074
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  10. 2019Gradient based sample selection for online continual learningcited by 568
  11. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  12. 2019On Tiny Episodic Memories in Continual Learningcited by 299
11 more
  1. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  2. 2018Selective Experience Replay for Lifelong Learningcited by 239
  3. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  4. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  5. 2019Episodic Memory in Lifelong Language Learningcited by 188
  6. 2016Achieving Human Parity in Conversational Speech Recognitioncited by 63
  7. 2020Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updatescited by 34
  8. 2018Continual Classification Learning Using Generative Modelscited by 28
  9. 2021Towards Lifelong Learning of End-to-end ASRcited by 20
  10. 2020Incremental Learning for End-to-End Automatic Speech Recognitioncited by 10
  11. 2019A Multi-Task Learning Framework for Overcoming the Catastrophic Forgetting in Automatic Speech Recognitioncited by 6

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 7; 2024: 3; 2025: 2.

15 papers.

Sort:

  1. 2022Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognitioncited by 10
  2. 2023Online Continual Learning in Keyword Spotting for Low-Resource Devices via Pooling High-Order Temporal Statisticscited by 9
  3. 2024Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whispercited by 9
  4. 2022An Investigation of the Combination of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language Understandingcited by 5
  5. 2022Continual Learning for On-Device Speech Recognition using Disentangled Conformerscited by 4
  6. 2023Improving Speech Recognition for African American English With Audio Classificationcited by 4
  7. 2025AnalyticKWS: Towards Exemplar-Free Analytic Class Incremental Learning for Small-footprint Keyword Spottingcited by 4
  8. 2023Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understandingcited by 3
  9. 2024Continuously Learning New Words in Automatic Speech Recognitioncited by 3
  10. 2024Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Modelcited by 2
  11. 2023Leveraging Cross-Utterance Context For ASR Decodingcited by 1
  12. 2023FusDom: Combining In-Domain and Out-of-Domain Knowledge for Continuous Self-Supervised Learningcited by 1
3 more
  1. 2023Conditional Online Learning for Keyword Spottingcited by 0
  2. 2023Continual Contrastive Spoken Language Understandingcited by 0
  3. 2025Navigating the Reality Gap: On-Device Continual Adaptation of ASR for Clinical Telephonycited by 0