Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model
Not on the map. Why?
Abstract (arXiv)
Recently, multi-task spoken language understanding (SLU) models have emerged, designed to address various speech processing tasks. However, these models often rely on a large number of parameters. Also, they often encounter difficulties in adapting to new data for a specific task without experiencing catastrophic forgetting of previously trained tasks. In this study, we propose finding task-specific subnetworks within a multi-task SLU model via neural network pruning. In addition to model compression, we expect that the forgetting of previously trained tasks can be mitigated by updating only a task-specific subnetwork. We conduct experiments on top of the state-of-the-art multi-task SLU model ``UniverSLU'', trained for several tasks such as emotion recognition (ER), intent classification (IC), and automatic speech recognition (ASR). We show that pruned models were successful in adapting to additional ASR or IC data with minimal performance degradation on previously trained tasks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 18 Jun 2024.
Builds on
30 of 37 references.
References · 30
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2016Learning without Forgettingcited by 2,496
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
- 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
- 2020What is the State of Neural Network Pruning?cited by 583
- 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
- 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
- 2021Generative Spoken Language Modeling from Raw Audiocited by 340
18 more
- 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
- 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
- 2024Continual Learning for Large Language Models: A Surveycited by 147
- 2019Continual Learning via Neural Pruningcited by 101
- 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
- 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
- 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
- 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 32
- 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
- 2018Spoken Language Understanding on the Edgecited by 27
- 2021Towards Lifelong Learning of End-to-end ASRcited by 21
- 2022Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognitioncited by 15
- 2022Online Continual Learning of End-to-End Speech Recognition Modelscited by 15
- 2021Omni-sparsity DNN: Fast Sparsity Optimization for On-Device Streaming E2E ASR via Supernetcited by 12
- 2022Learning ASR pathways: A sparse multilingual ASR modelcited by 12
- 2022Continual Learning for On-Device Speech Recognition using Disentangled Conformerscited by 4
- 2022Updating Only Encoders Prevents Catastrophic Forgetting of End-to-End ASR Modelscited by 3
- 2023Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understandingcited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.