# CrossAligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding

arXiv: https://arxiv.org/abs/2203.09982 · HTML: https://mlsanity.com/paper/2203.09982/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog](https://mlsanity.com/paper/1810.13327/) (2018)
- [End-to-End Slot Alignment and Recognition for Cross-Lingual NLU](https://mlsanity.com/paper/2004.14353/) (2020)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [The Missing Ingredient in Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/1903.07091/) (2019)
- [Multi-Loss Weighting with Coefficient of Variations](https://mlsanity.com/paper/2009.01717/) (2020)
- [Entity Projection via Machine Translation for Cross-Lingual NER](https://mlsanity.com/paper/1909.05356/) (2019)
- [Multilingual BERT Post-Pretraining Alignment](https://mlsanity.com/paper/2010.12547/) (2020)
- [Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1909.00437/) (2019)
- [MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning](https://mlsanity.com/paper/2104.07908/) (2021)
- [DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries](https://mlsanity.com/paper/2010.12566/) (2020)
- [Rethinking InfoNCE: How Many Negative Samples Do You Need?](https://mlsanity.com/paper/2105.13003/) (2021)
- [Conversation Graph: Data Augmentation, Training and Evaluation for Non-Deterministic Dialogue Management](https://mlsanity.com/paper/2010.15411/) (2020)
- [XeroAlign: Zero-Shot Cross-lingual Transformer Alignment](https://mlsanity.com/paper/2105.02472/) (2021)
- [Cross-Lingual Named Entity Recognition Using Parallel Corpus: A New Approach Using XLM-RoBERTa Alignment](https://mlsanity.com/paper/2101.11112/) (2021)
- [A Benchmarking Environment for Reinforcement Learning Based Task Oriented Dialogue Management](https://mlsanity.com/paper/1711.11023/) (2017)

## Built on by (most cited first, 7 of 7)

- [Transfer-Free Data-Efficient Multilingual Slot Labeling](https://mlsanity.com/paper/2305.13528/) (2023)
- [Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding](https://mlsanity.com/paper/2205.03656/) (2022)
- [Bootstrapping Multilingual Semantic Parsers using Large Language Models](https://mlsanity.com/paper/2210.07313/) (2022)
- [Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing](https://mlsanity.com/paper/2307.04096/) (2023)
- [Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunities](https://mlsanity.com/paper/2210.16960/) (2022)
- [Towards Zero-Shot Functional Compositionality of Language Models](https://mlsanity.com/paper/2303.03103/) (2023)
- [LaDA: Latent Dialogue Action For Zero-shot Cross-lingual Neural Network Language Modeling](https://mlsanity.com/paper/2308.02903/) (2023)

## Abstract (arXiv)

Task-oriented personal assistants enable people to interact with a host of devices and services using natural language. One of the challenges of making neural dialogue systems available to more users is the lack of training data for all but a few languages. Zero-shot methods try to solve this issue by acquiring task knowledge in a high-resource language such as English with the aim of transferring it to the low-resource language(s). To this end, we introduce CrossAligner, the principal method of a variety of effective approaches for zero-shot cross-lingual transfer based on learning alignment from unlabelled parallel data. We present a quantitative analysis of individual methods as well as their weighted combinations, several of which exceed state-of-the-art (SOTA) scores as evaluated across nine languages, fifteen test sets and three benchmark multilingual datasets. A detailed qualitative error analysis of the best methods shows that our fine-tuned language models can zero-shot transfer the task knowledge better than anticipated.