# I2KD-SLU: An Intra-Inter Knowledge Distillation Framework for Zero-Shot Cross-Lingual Spoken Language Understanding

arXiv: https://arxiv.org/abs/2310.02594 · HTML: https://mlsanity.com/paper/2310.02594/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog](https://mlsanity.com/paper/1810.13327/) (2018)
- [Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling](https://mlsanity.com/paper/2301.03580/) (2023)
- [End-to-End Slot Alignment and Recognition for Cross-Lingual NLU](https://mlsanity.com/paper/2004.14353/) (2020)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [A Co-Interactive Transformer for Joint Slot Filling and Intent Detection](https://mlsanity.com/paper/2010.03880/) (2020)
- [Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2001.11458/) (2020)
- [Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing](https://mlsanity.com/paper/1909.06775/) (2019)
- [GL-GIN: Fast and Accurate Non-Autoregressive Model for Joint Multiple Intent Detection and Slot Filling](https://mlsanity.com/paper/2106.01925/) (2021)
- [Improving Auto-Augment via Augmentation-Wise Weight Sharing](https://mlsanity.com/paper/2009.14737/) (2020)
- [Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation](https://mlsanity.com/paper/2303.15932/) (2023)
- [M3ST: Mix at Three Levels for Speech Translation](https://mlsanity.com/paper/2212.03657/) (2022)
- [PolyVoice: Language Models for Speech to Speech Translation](https://mlsanity.com/paper/2306.02982/) (2023)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [PoseRAC: Pose Saliency Transformer for Repetitive Action Counting](https://mlsanity.com/paper/2303.08450/) (2023)
- [GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding](https://mlsanity.com/paper/2204.08325/) (2022)
- [Co-guiding Net: Achieving Mutual Guidances between Multiple Intent Detection and Slot Filling via Heterogeneous Semantics-Label Graphs](https://mlsanity.com/paper/2210.10375/) (2022)
- [Don't Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding](https://mlsanity.com/paper/2010.03714/) (2020)
- [SSVMR: Saliency-based Self-training for Video-Music Retrieval](https://mlsanity.com/paper/2302.09328/) (2023)
- [PIN: A Novel Parallel Interactive Network for Spoken Language Understanding](https://mlsanity.com/paper/2009.13431/) (2020)
- [Towards Joint Intent Detection and Slot Filling via Higher-order Attention](https://mlsanity.com/paper/2109.08890/) (2021)
- [Powering One-shot Topological NAS with Stabilized Share-parameter Proxy](https://mlsanity.com/paper/2005.10511/) (2020)
- [A Dynamic Graph Interactive Framework with Label-Semantic Injection for Spoken Language Understanding](https://mlsanity.com/paper/2211.04023/) (2022)
- [Using Neighborhood Context to Improve Information Extraction from Visual Documents Captured on Mobile Phones](https://mlsanity.com/paper/2108.10395/) (2021)

## Abstract (arXiv)

Spoken language understanding (SLU) typically includes two subtasks: intent detection and slot filling. Currently, it has achieved great success in high-resource languages, but it still remains challenging in low-resource languages due to the scarcity of labeled training data. Hence, there is a growing interest in zero-shot cross-lingual SLU. Despite of the success of existing zero-shot cross-lingual SLU models, most of them neglect to achieve the mutual guidance between intent and slots. To address this issue, we propose an Intra-Inter Knowledge Distillation framework for zero-shot cross-lingual Spoken Language Understanding (I2KD-SLU) to model the mutual guidance. Specifically, we not only apply intra-knowledge distillation between intent predictions or slot predictions of the same utterance in different languages, but also apply inter-knowledge distillation between intent predictions and slot predictions of the same utterance. Our experimental results demonstrate that our proposed framework significantly improves the performance compared with the strong baselines and achieves the new state-of-the-art performance on the MultiATIS++ dataset, obtaining a significant improvement over the previous best model in overall accuracy.