# Multitask Learning for Low Resource Spoken Language Understanding

arXiv: https://arxiv.org/abs/2211.13703 · HTML: https://mlsanity.com/paper/2211.13703/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 24 Nov 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [Transformers with convolutional context for ASR](https://mlsanity.com/paper/1904.11660/) (2019)
- [Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces](https://mlsanity.com/paper/2008.06173/) (2020)
- [Capsule Networks for Low Resource Spoken Language Understanding](https://mlsanity.com/paper/1805.02922/) (2018)
- [Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding](https://mlsanity.com/paper/2106.15065/) (2021)

## Built on by (most cited first, 3 of 3)

- [Pre-Finetuning for Few-Shot Emotional Speech Recognition](https://mlsanity.com/paper/2302.12921/) (2023)
- [Few-Shot Spoken Language Understanding via Joint Speech-Text Models](https://mlsanity.com/paper/2310.05919/) (2023)
- [An Adapter-Based Unified Model for Multiple Spoken Language Processing Tasks](https://mlsanity.com/paper/2406.14747/) (2024)

## Abstract (arXiv)

We explore the benefits that multitask learning offer to speech processing as we train models on dual objectives with automatic speech recognition and intent classification or sentiment classification. Our models, although being of modest size, show improvements over models trained end-to-end on intent classification. We compare different settings to find the optimal disposition of each task module compared to one another. Finally, we study the performance of the models in low-resource scenario by training the models with as few as one example per class. We show that multitask learning in these scenarios compete with a baseline model trained on text features and performs considerably better than a pipeline model. On sentiment classification, we match the performance of an end-to-end model with ten times as many parameters. We consider 4 tasks and 4 datasets in Dutch and English.