# Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2203.16082 · HTML: https://mlsanity.com/paper/2203.16082/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2022.

## Built on by (most cited first, 15 of 15)

- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Learning from models beyond fine-tuning](https://mlsanity.com/paper/2310.08184/) (2023)
- [Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition](https://mlsanity.com/paper/2210.15282/) (2022)
- [MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval](https://mlsanity.com/paper/2309.01516/) (2023)
- [Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper](https://mlsanity.com/paper/2408.10680/) (2024)
- [Low-rank Adaptation Method for Wav2vec2-based Fake Audio Detection](https://mlsanity.com/paper/2306.05617/) (2023)
- [Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding](https://mlsanity.com/paper/2303.03267/) (2023)
- [Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model](https://mlsanity.com/paper/2406.12317/) (2024)
- [NIRANTAR: Continual Learning with New Languages and Domains on Real-world Speech Data](https://mlsanity.com/paper/2507.00534/) (2025)
- [Damage Control During Domain Adaptation for Transducer Based Automatic Speech Recognition](https://mlsanity.com/paper/2210.03255/) (2022)
- [Decoupled Prompt-Adapter Tuning for Continual Activity Recognition](https://mlsanity.com/paper/2407.14811/) (2024)
- [Continual Learning With Quasi-Newton Methods](https://mlsanity.com/paper/2503.19939/) (2025)
- [Audio-AdapterFusion: A Task-ID-free Approach for Efficient and Non-Destructive Multi-task Speech Recognition](https://mlsanity.com/paper/2310.13015/) (2023)
- [Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASR](https://mlsanity.com/paper/2607.11163/) (2026)
- [Traceback Translators Against Forgetting in Continual Fake Speech Detection](https://mlsanity.com/paper/2607.12569/) (2026)

## Abstract (arXiv)

Learning a set of tasks in sequence remains a challenge for artificial neural networks, which, in such scenarios, tend to suffer from Catastrophic Forgetting (CF). The same applies to End-to-End (E2E) Automatic Speech Recognition (ASR) models, even for monolingual tasks. In this paper, we aim to overcome CF for E2E ASR by inserting adapters, small architectures of few parameters which allow a general model to be fine-tuned to a specific task, into our model. We make these adapters task-specific, while regularizing the parameters of the model shared by all tasks, thus stimulating the model to fully exploit the adapters while keeping the shared parameters to work well for all tasks. Our method outperforms all baselines on two monolingual experiments while being more storage efficient and without requiring the storage of data from previous tasks.