# An Adapter-Based Unified Model for Multiple Spoken Language Processing Tasks

arXiv: https://arxiv.org/abs/2406.14747 · HTML: https://mlsanity.com/paper/2406.14747/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 31 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [A Co-Interactive Transformer for Joint Slot Filling and Intent Detection](https://mlsanity.com/paper/2010.03880/) (2020)
- [Lightweight Adapter Tuning for Multilingual Speech Translation](https://mlsanity.com/paper/2106.01463/) (2021)
- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks](https://mlsanity.com/paper/2010.11338/) (2020)
- [Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.03218/) (2022)
- [Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?](https://mlsanity.com/paper/2306.00452/) (2023)
- [Fusing ASR Outputs in Joint Training for Speech Emotion Recognition](https://mlsanity.com/paper/2110.15684/) (2021)
- [SpeechNet: A Universal Modularized Model for Speech Processing Tasks](https://mlsanity.com/paper/2105.03070/) (2021)
- [NAVER LABS Europe's Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Track](https://mlsanity.com/paper/2306.07763/) (2023)
- [Towards Semi-Supervised Semantics Understanding from Speech](https://mlsanity.com/paper/2011.06195/) (2020)
- [Multi-task RNN-T with Semantic Decoder for Streamable Spoken Language Understanding](https://mlsanity.com/paper/2204.00558/) (2022)
- [Multitask Learning for Low Resource Spoken Language Understanding](https://mlsanity.com/paper/2211.13703/) (2022)

## Abstract (arXiv)

Self-supervised learning models have revolutionized the field of speech processing. However, the process of fine-tuning these models on downstream tasks requires substantial computational resources, particularly when dealing with multiple speech-processing tasks. In this paper, we explore the potential of adapter-based fine-tuning in developing a unified model capable of effectively handling multiple spoken language processing tasks. The tasks we investigate are Automatic Speech Recognition, Phoneme Recognition, Intent Classification, Slot Filling, and Spoken Emotion Recognition. We validate our approach through a series of experiments on the SUPERB benchmark, and our results indicate that adapter-based fine-tuning enables a single encoder-decoder model to perform multiple speech processing tasks with an average improvement of 18.4% across the five target tasks while staying efficient in terms of parameter updates.