# Continual Learning for Monolingual End-to-End Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2112.09427 · HTML: https://mlsanity.com/paper/2112.09427/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [SynthASR: Unlocking Synthetic Data for Speech Recognition](https://mlsanity.com/paper/2106.07803/) (2021)
- [Towards Lifelong Learning of End-to-end ASR](https://mlsanity.com/paper/2104.01616/) (2021)
- [Gradient Episodic Memory with a Soft Constraint for Continual Learning](https://mlsanity.com/paper/2011.07801/) (2020)
- [Incremental Learning for End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2005.04288/) (2020)
- [Batch-level Experience Replay with Review for Continual Learning](https://mlsanity.com/paper/2007.05683/) (2020)

## Built on by (most cited first, 8 of 8)

- [Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition](https://mlsanity.com/paper/2210.15282/) (2022)
- [Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper](https://mlsanity.com/paper/2408.10680/) (2024)
- [Continual Learning for On-Device Speech Recognition using Disentangled Conformers](https://mlsanity.com/paper/2212.01393/) (2022)
- [Replay to Remember: Continual Layer-Specific Fine-tuning for German Speech Recognition](https://mlsanity.com/paper/2307.07280/) (2023)
- [Continuously Learning New Words in Automatic Speech Recognition](https://mlsanity.com/paper/2401.04482/) (2024)
- [A Parameter-efficient Language Extension Framework for Multilingual ASR](https://mlsanity.com/paper/2406.06329/) (2024)
- [Code-Switched Urdu ASR for Noisy Telephonic Environment using Data Centric Approach with Hybrid HMM and CNN-TDNN](https://mlsanity.com/paper/2307.12759/) (2023)
- [Damage Control During Domain Adaptation for Transducer Based Automatic Speech Recognition](https://mlsanity.com/paper/2210.03255/) (2022)

## Abstract (arXiv)

Adapting Automatic Speech Recognition (ASR) models to new domains results in a deterioration of performance on the original domain(s), a phenomenon called Catastrophic Forgetting (CF). Even monolingual ASR models cannot be extended to new accents, dialects, topics, etc. without suffering from CF, making them unable to be continually enhanced without storing all past data. Fortunately, Continual Learning (CL) methods, which aim to enable continual adaptation while overcoming CF, can be used. In this paper, we implement an extensive number of CL methods for End-to-End ASR and test and compare their ability to extend a monolingual Hybrid CTC-Transformer model across four new tasks. We find that the best performing CL method closes the gap between the fine-tuned model (lower bound) and the model trained jointly on all tasks (upper bound) by more than 40%, while requiring access to only 0.6% of the original data.