# Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding

arXiv: https://arxiv.org/abs/2205.03656 · HTML: https://mlsanity.com/paper/2205.03656/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 7 May 2022.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog](https://mlsanity.com/paper/1810.13327/) (2018)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019)
- [Learned in Translation: Contextualized Word Vectors](https://mlsanity.com/paper/1708.00107/) (2017)
- [End-to-End Slot Alignment and Recognition for Cross-Lingual NLU](https://mlsanity.com/paper/2004.14353/) (2020)
- [Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network](https://mlsanity.com/paper/2006.05702/) (2020)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [A Co-Interactive Transformer for Joint Slot Filling and Intent Detection](https://mlsanity.com/paper/2010.03880/) (2020)
- [On Learning Universal Representations Across Languages](https://mlsanity.com/paper/2007.15960/) (2020)
- [From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding](https://mlsanity.com/paper/2105.07316/) (2021)
- [Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systems](https://mlsanity.com/paper/1911.09273/) (2019)
- [SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/2010.02693/) (2020)
- [Zero-shot Cross-lingual Dialogue Systems with Transferable Latent Variables](https://mlsanity.com/paper/1911.04081/) (2019)
- [GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding](https://mlsanity.com/paper/2204.08325/) (2022)
- [XeroAlign: Zero-Shot Cross-lingual Transformer Alignment](https://mlsanity.com/paper/2105.02472/) (2021)
- [CrossAligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding](https://mlsanity.com/paper/2203.09982/) (2022)
- [Cross-lingual Spoken Language Understanding with Regularized Representation Alignment](https://mlsanity.com/paper/2009.14510/) (2020)
- [An Effective Non-Autoregressive Model for Spoken Language Understanding](https://mlsanity.com/paper/2108.07005/) (2021)

## Built on by (most cited first, 3 of 3)

- [Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing](https://mlsanity.com/paper/2307.04096/) (2023)
- [I2KD-SLU: An Intra-Inter Knowledge Distillation Framework for Zero-Shot Cross-Lingual Spoken Language Understanding](https://mlsanity.com/paper/2310.02594/) (2023)
- [HC2L: Hybrid and Cooperative Contrastive Learning for Cross-lingual Spoken Language Understanding](https://mlsanity.com/paper/2405.06204/) (2024)

## Abstract (arXiv)

Despite the great success of spoken language understanding (SLU) in high-resource languages, it remains challenging in low-resource languages mainly due to the lack of labeled training data. The recent multilingual code-switching approach achieves better alignments of model representations across languages by constructing a mixed-language context in zero-shot cross-lingual SLU. However, current code-switching methods are limited to implicit alignment and disregard the inherent semantic structure in SLU, i.e., the hierarchical inclusion of utterances, slots, and words. In this paper, we propose to model the utterance-slot-word structure by a multi-level contrastive learning framework at the utterance, slot, and word levels to facilitate explicit alignment. Novel code-switching schemes are introduced to generate hard negative examples for our contrastive learning framework. Furthermore, we develop a label-aware joint model leveraging label semantics to enhance the implicit alignment and feed to contrastive learning. Our experimental results show that our proposed methods significantly improve the performance compared with the strong baselines on two zero-shot cross-lingual SLU benchmark datasets.