# Audio-AdapterFusion: A Task-ID-free Approach for Efficient and Non-Destructive Multi-task Speech Recognition

arXiv: https://arxiv.org/abs/2310.13015 · HTML: https://mlsanity.com/paper/2310.13015/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://mlsanity.com/paper/1811.01088/) (2018)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks](https://mlsanity.com/paper/2010.11338/) (2020)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.03218/) (2022)
- [Transfer Learning for Speech Recognition on a Budget](https://mlsanity.com/paper/1706.00290/) (2017)
- [Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2203.16082/) (2022)
- [E2E-based Multi-task Learning Approach to Joint Speech and Accent Recognition](https://mlsanity.com/paper/2106.08211/) (2021)
- [Transformer-Based Models for Question Answering on COVID19](https://mlsanity.com/paper/2101.11432/) (2021)
- [Modular Domain Adaptation for Conformer-Based Streaming ASR](https://mlsanity.com/paper/2305.13408/) (2023)
- [An Experimental Evaluation of Transformer-based Language Models in the Biomedical Domain](https://mlsanity.com/paper/2012.15419/) (2020)
- [Doctor XAvIer: Explainable Diagnosis on Physician-Patient Dialogues and XAI Evaluation](https://mlsanity.com/paper/2204.10178/) (2022)
- [A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale](https://mlsanity.com/paper/2304.11053/) (2023)

## Abstract (arXiv)

Adapters are an efficient, composable alternative to full fine-tuning of pre-trained models and help scale the deployment of large ASR models to many tasks. In practice, a task ID is commonly prepended to the input during inference to route to single-task adapters for the specified task. However, one major limitation of this approach is that the task ID may not be known during inference, rendering it unsuitable for most multi-task settings. To address this, we propose three novel task-ID-free methods to combine single-task adapters in multi-task ASR and investigate two learning algorithms for training. We evaluate our methods on 10 test sets from 4 diverse ASR tasks and show that our methods are non-destructive and parameter-efficient. While only updating 17% of the model parameters, our methods can achieve an 8% mean WER improvement relative to full fine-tuning and are on-par with task-ID adapter routing.