# Joint Multiple Intent Detection and Slot Filling via Self-distillation

arXiv: https://arxiv.org/abs/2108.08042 · HTML: https://mlsanity.com/paper/2108.08042/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 18 Aug 2021.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016)
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019)
- [A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1907.00390/) (2019)
- [A Bi-model based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling](https://mlsanity.com/paper/1812.10235/) (2018)
- [PIN: A Novel Parallel Interactive Network for Spoken Language Understanding](https://mlsanity.com/paper/2009.13431/) (2020)

## Built on by (most cited first, 12 of 12)

- [A Dynamic Graph Interactive Framework with Label-Semantic Injection for Spoken Language Understanding](https://mlsanity.com/paper/2211.04023/) (2022)
- [Joint Multiple Intent Detection and Slot Filling with Supervised Contrastive Learning and Self-Distillation](https://mlsanity.com/paper/2308.14654/) (2023)
- [MISCA: A Joint Model for Multiple Intent Detection and Slot Filling with Intent-Slot Co-Attention](https://mlsanity.com/paper/2312.05741/) (2023)
- [ECLM: Entity Level Language Model for Spoken Language Understanding with Chain of Intent](https://mlsanity.com/paper/2403.04481/) (2024)
- [A Unified Framework for Multi-intent Spoken Language Understanding with prompting](https://mlsanity.com/paper/2210.03337/) (2022)
- [JPIS: A Joint Model for Profile-based Intent Detection and Slot Filling with Slot-to-Intent Attention](https://mlsanity.com/paper/2312.08737/) (2023)
- [A Two-Stage Prediction-Aware Contrastive Learning Framework for Multi-Intent NLU](https://mlsanity.com/paper/2405.02925/) (2024)
- [MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU](https://mlsanity.com/paper/2408.08144/) (2024)
- [Calibrate and Refine! A Novel and Agile Framework for ASR-error Robust Intent Detection](https://mlsanity.com/paper/2205.11008/) (2022)
- [Feature-augmented Machine Reading Comprehension with Auxiliary Tasks](https://mlsanity.com/paper/2211.09438/) (2022)
- [Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning](https://mlsanity.com/paper/2405.20852/) (2024)
- [Multi-Intent Spoken Language Understanding: Methods, Trends, and Challenges](https://mlsanity.com/paper/2512.11258/) (2025)

## Abstract (arXiv)

Intent detection and slot filling are two main tasks in natural language understanding (NLU) for identifying users' needs from their utterances. These two tasks are highly related and often trained jointly. However, most previous works assume that each utterance only corresponds to one intent, ignoring the fact that a user utterance in many cases could include multiple intents. In this paper, we propose a novel Self-Distillation Joint NLU model (SDJN) for multi-intent NLU. First, we formulate multiple intent detection as a weakly supervised problem and approach with multiple instance learning (MIL). Then, we design an auxiliary loop via self-distillation with three orderly arranged decoders: Initial Slot Decoder, MIL Intent Decoder, and Final Slot Decoder. The output of each decoder will serve as auxiliary information for the next decoder. With the auxiliary knowledge provided by the MIL Intent Decoder, we set Final Slot Decoder as the teacher model that imparts knowledge back to Initial Slot Decoder to complete the loop. The auxiliary loop enables intents and slots to guide mutually in-depth and further boost the overall NLU performance. Experimental results on two public multi-intent datasets indicate that our model achieves strong performance compared to others.