# Incremental Learning for End-to-End Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2005.04288 · HTML: https://mlsanity.com/paper/2005.04288/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2020.

## Built on by (most cited first, 10 of 10)

- [Towards Lifelong Learning of End-to-end ASR](https://mlsanity.com/paper/2104.01616/) (2021)
- [Online Continual Learning of End-to-End Speech Recognition Models](https://mlsanity.com/paper/2207.05071/) (2022)
- [Continual Learning for Monolingual End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2112.09427/) (2021)
- [Progressive Continual Learning for Spoken Keyword Spotting](https://mlsanity.com/paper/2201.12546/) (2022)
- [Alternative Pseudo-Labeling for Semi-Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2308.06547/) (2023)
- [Leveraging Label Information for Multimodal Emotion Recognition](https://mlsanity.com/paper/2309.02106/) (2023)
- [Continuously Learning New Words in Automatic Speech Recognition](https://mlsanity.com/paper/2401.04482/) (2024)
- [SCaLa: Supervised Contrastive Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.04187/) (2021)
- [UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognition](https://mlsanity.com/paper/2412.17507/) (2024)
- [A Multi-tasking Model of Speaker-Keyword Classification for Keeping Human in the Loop of Drone-assisted Inspection](https://mlsanity.com/paper/2207.04027/) (2022)

## Abstract (arXiv)

In this paper, we propose an incremental learning method for end-to-end Automatic Speech Recognition (ASR) which enables an ASR system to perform well on new tasks while maintaining the performance on its originally learned ones. To mitigate catastrophic forgetting during incremental learning, we design a novel explainability-based knowledge distillation for ASR models, which is combined with a response-based knowledge distillation to maintain the original model's predictions and the "reason" for the predictions. Our method works without access to the training data of original tasks, which addresses the cases where the previous data is no longer available or joint training is costly. Results on a multi-stage sequential training task show that our method outperforms existing ones in mitigating forgetting. Furthermore, in two practical scenarios, compared to the target-reference joint training method, the performance drop of our method is 0.02% Character Error Rate (CER), which is 97% smaller than the drops of the baseline methods.