mlsanityData to 7 Oct 2026

MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU

arXiv 2408.08144 ↗First posted 15 Aug 2024cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Although Large Language Models (LLMs) can generate coherent text, they often struggle to recognise user intent behind queries. In contrast, Natural Language Understanding (NLU) models interpret the purpose and key information of user input for responsive interactions. Existing NLU models typically map utterances to a dual-level semantic frame, involving sentence-level intent (SI) and word-level slot (WS) labels. However, real-life conversations primarily consist of multi-turn dialogues, requiring the interpretation of complex and extended exchanges. Researchers encounter challenges in addressing all facets of multi-turn dialogue using a unified NLU model. This paper introduces MIDAS, a novel approach leveraging multi-level intent, domain, and slot knowledge distillation for multi-turn NLU. We construct distinct teachers for SI detection, WS filling, and conversation-level domain (CD) classification, each fine-tuned for specific knowledge. A multi-teacher loss is proposed to facilitate the integration of these teachers, guiding a student model in multi-turn dialogue tasks. Results demonstrate the efficacy of our model in improving multi-turn conversation understanding, showcasing the potential for advancements in NLU through multi-level dialogue knowledge distillation. Our implementation is open-sourced on https://github.com/adlnlp/Midas.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 38 of its 56 references that are in the map. First posted on arXiv 15 Aug 2024.

Builds on

38 of 56 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  8. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 201
  9. 2019BERT for Joint Intent Classification and Slot Fillingcited by 195
  10. 2016Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Fillingcited by 137
  11. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 82
  12. 2020Reinforced Multi-Teacher Selection for Knowledge Distillationcited by 62
26 more
  1. 2020A Co-Interactive Transformer for Joint Slot Filling and Intent Detectioncited by 61
  2. 2018Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systemscited by 53
  3. 2021One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teacherscited by 39
  4. 2020Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domainscited by 33
  5. 2021GL-GIN: Fast and Accurate Non-Autoregressive Model for Joint Multiple Intent Detection and Slot Fillingcited by 33
  6. 2021Cross-modal Knowledge Distillation for Vision-to-Sensor Action Recognitioncited by 25
  7. 2021Intent Detection and Slot Filling for Vietnamesecited by 19
  8. 2020Exploring Transfer Learning For End-to-End Spoken Language Understandingcited by 17
  9. 2021Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classificationcited by 14
  10. 2021Joint Multiple Intent Detection and Slot Filling via Self-distillationcited by 12
  11. 2017Sequential Dialogue Context Modeling for Spoken Language Understandingcited by 11
  12. 2021Towards Joint Intent Detection and Slot Filling via Higher-order Attentioncited by 11
  13. 2020Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Fillingcited by 10
  14. 2021X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answeringcited by 10
  15. 2022Group is better than individual: Exploiting Label Topologies and Label Relations for Joint Multiple Intent Detection and Slot Fillingcited by 10
  16. 2023Ensemble knowledge distillation of self-supervised speech modelscited by 10
  17. 2022Bi-directional Joint Neural Networks for Intent Classification and Slot Fillingcited by 8
  18. 2023Joint Multiple Intent Detection and Slot Filling with Supervised Contrastive Learning and Self-Distillationcited by 7
  19. 2023MISCA: A Joint Model for Multiple Intent Detection and Slot Filling with Intent-Slot Co-Attentioncited by 7
  20. 2021A Context-Aware Hierarchical BERT Fusion Network for Multi-turn Dialog Act Detectioncited by 5
  21. 2022Explainable Slot Type Attentions to Improve Joint Intent Detection and Slot Fillingcited by 5
  22. 2022mcBERT: Momentum Contrastive Learning with BERT for Zero-Shot Slot Fillingcited by 4
  23. 2023Towards a Unified Conversational Recommendation System: Multi-task Learning via Contextualized Knowledge Distillationcited by 4
  24. 2022A Fast Attention Network for Joint Intent Detection and Slot Filling on Edge Devicescited by 3
  25. 2024A Two-Stage Prediction-Aware Contrastive Learning Framework for Multi-Intent NLUcited by 2
  26. 2023Tri-level Joint Natural Language Understanding for Multi-turn Conversational Datasetscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024ChuLo: Chunk-Level Key Information Representation for Long Document Understandingcited by 2
  2. 2025AFD-SLU: Adaptive Feature Distillation for Spoken Language Understandingcited by 1