# MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU

arXiv: https://arxiv.org/abs/2408.08144 · HTML: https://mlsanity.com/paper/2408.08144/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 38 of its 56 references that are in the map.
- First posted on arXiv 15 Aug 2024.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 30 of 38)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016)
- [Densely Guided Knowledge Distillation using Multiple Teacher Assistants](https://mlsanity.com/paper/2009.08825/) (2020)
- [Reinforced Multi-Teacher Selection for Knowledge Distillation](https://mlsanity.com/paper/2012.06048/) (2020)
- [A Co-Interactive Transformer for Joint Slot Filling and Intent Detection](https://mlsanity.com/paper/2010.03880/) (2020)
- [Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1804.06512/) (2018)
- [One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers](https://mlsanity.com/paper/2106.01023/) (2021)
- [Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains](https://mlsanity.com/paper/2012.01266/) (2020)
- [GL-GIN: Fast and Accurate Non-Autoregressive Model for Joint Multiple Intent Detection and Slot Filling](https://mlsanity.com/paper/2106.01925/) (2021)
- [Cross-modal Knowledge Distillation for Vision-to-Sensor Action Recognition](https://mlsanity.com/paper/2112.01849/) (2021)
- [Intent Detection and Slot Filling for Vietnamese](https://mlsanity.com/paper/2104.02021/) (2021)
- [Exploring Transfer Learning For End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2012.08549/) (2020)
- [Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification](https://mlsanity.com/paper/2108.02598/) (2021)
- [Joint Multiple Intent Detection and Slot Filling via Self-distillation](https://mlsanity.com/paper/2108.08042/) (2021)
- [Ensemble knowledge distillation of self-supervised speech models](https://mlsanity.com/paper/2302.12757/) (2023)
- [Sequential Dialogue Context Modeling for Spoken Language Understanding](https://mlsanity.com/paper/1705.03455/) (2017)
- [Towards Joint Intent Detection and Slot Filling via Higher-order Attention](https://mlsanity.com/paper/2109.08890/) (2021)
- [Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling](https://mlsanity.com/paper/2012.11689/) (2020)
- [X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answering](https://mlsanity.com/paper/2104.09696/) (2021)
- [Group is better than individual: Exploiting Label Topologies and Label Relations for Joint Multiple Intent Detection and Slot Filling](https://mlsanity.com/paper/2210.10369/) (2022)
- [Bi-directional Joint Neural Networks for Intent Classification and Slot Filling](https://mlsanity.com/paper/2202.13079/) (2022)
- [Joint Multiple Intent Detection and Slot Filling with Supervised Contrastive Learning and Self-Distillation](https://mlsanity.com/paper/2308.14654/) (2023)

## Built on by (most cited first, 2 of 2)

- [ChuLo: Chunk-Level Key Information Representation for Long Document Understanding](https://mlsanity.com/paper/2410.11119/) (2024)
- [AFD-SLU: Adaptive Feature Distillation for Spoken Language Understanding](https://mlsanity.com/paper/2509.04821/) (2025)

## Abstract (arXiv)

Although Large Language Models (LLMs) can generate coherent text, they often struggle to recognise user intent behind queries. In contrast, Natural Language Understanding (NLU) models interpret the purpose and key information of user input for responsive interactions. Existing NLU models typically map utterances to a dual-level semantic frame, involving sentence-level intent (SI) and word-level slot (WS) labels. However, real-life conversations primarily consist of multi-turn dialogues, requiring the interpretation of complex and extended exchanges. Researchers encounter challenges in addressing all facets of multi-turn dialogue using a unified NLU model. This paper introduces MIDAS, a novel approach leveraging multi-level intent, domain, and slot knowledge distillation for multi-turn NLU. We construct distinct teachers for SI detection, WS filling, and conversation-level domain (CD) classification, each fine-tuned for specific knowledge. A multi-teacher loss is proposed to facilitate the integration of these teachers, guiding a student model in multi-turn dialogue tasks. Results demonstrate the efficacy of our model in improving multi-turn conversation understanding, showcasing the potential for advancements in NLU through multi-level dialogue knowledge distillation. Our implementation is open-sourced on https://github.com/adlnlp/Midas.