# Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model

arXiv: https://arxiv.org/abs/2406.12317 · HTML: https://mlsanity.com/paper/2406.12317/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [Continual Learning via Neural Pruning](https://mlsanity.com/paper/1903.04476/) (2019)
- [LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT](https://mlsanity.com/paper/2310.04673/) (2023)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks](https://mlsanity.com/paper/2303.00733/) (2023)
- [Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding](https://mlsanity.com/paper/2302.14132/) (2023)
- [Spoken Language Understanding on the Edge](https://mlsanity.com/paper/1810.12735/) (2018)
- [Towards Lifelong Learning of End-to-end ASR](https://mlsanity.com/paper/2104.01616/) (2021)
- [Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2203.16082/) (2022)
- [Online Continual Learning of End-to-End Speech Recognition Models](https://mlsanity.com/paper/2207.05071/) (2022)
- [Omni-sparsity DNN: Fast Sparsity Optimization for On-Device Streaming E2E ASR via Supernet](https://mlsanity.com/paper/2110.08352/) (2021)
- [Learning ASR pathways: A sparse multilingual ASR model](https://mlsanity.com/paper/2209.05735/) (2022)
- [Continual Learning for On-Device Speech Recognition using Disentangled Conformers](https://mlsanity.com/paper/2212.01393/) (2022)
- [Updating Only Encoders Prevents Catastrophic Forgetting of End-to-End ASR Models](https://mlsanity.com/paper/2207.00216/) (2022)
- [Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2305.13899/) (2023)

## Built on by (most cited first, 2 of 2)

- [Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering](https://mlsanity.com/paper/2603.06854/) (2026)
- [Developing a High-performance Framework for Speech Emotion Recognition in Naturalistic Conditions Challenge for Emotional Attribute Prediction](https://mlsanity.com/paper/2506.10930/) (2025)

## Abstract (arXiv)

Recently, multi-task spoken language understanding (SLU) models have emerged, designed to address various speech processing tasks. However, these models often rely on a large number of parameters. Also, they often encounter difficulties in adapting to new data for a specific task without experiencing catastrophic forgetting of previously trained tasks. In this study, we propose finding task-specific subnetworks within a multi-task SLU model via neural network pruning. In addition to model compression, we expect that the forgetting of previously trained tasks can be mitigated by updating only a task-specific subnetwork. We conduct experiments on top of the state-of-the-art multi-task SLU model ``UniverSLU'', trained for several tasks such as emotion recognition (ER), intent classification (IC), and automatic speech recognition (ASR). We show that pruned models were successful in adapting to additional ASR or IC data with minimal performance degradation on previously trained tasks.