# From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

arXiv: https://arxiv.org/abs/2105.07316 · HTML: https://mlsanity.com/paper/2105.07316/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 15 May 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog](https://mlsanity.com/paper/1810.13327/) (2018)
- [Offline bilingual word vectors, orthogonal transformations and the inverted softmax](https://mlsanity.com/paper/1702.03859/) (2017)
- [End-to-End Slot Alignment and Recognition for Cross-Lingual NLU](https://mlsanity.com/paper/2004.14353/) (2020)
- [A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks](https://mlsanity.com/paper/1611.01587/) (2016)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too](https://mlsanity.com/paper/2005.13013/) (2020)
- [Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Model](https://mlsanity.com/paper/1907.02884/) (2019)
- [On the use of BERT for Neural Machine Translation](https://mlsanity.com/paper/1909.12744/) (2019)
- [Zero-shot Cross-lingual Dialogue Systems with Transferable Latent Variables](https://mlsanity.com/paper/1911.04081/) (2019)
- [Almawave-SLU: A new dataset for SLU in Italian](https://mlsanity.com/paper/1907.07526/) (2019)
- [El Volumen Louder Por Favor: Code-switching in Task-oriented Semantic Parsing](https://mlsanity.com/paper/2101.10524/) (2021)
- [Don't Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddings](https://mlsanity.com/paper/2004.15001/) (2020)

## Built on by (most cited first, 30 of 43)

- [A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods](https://mlsanity.com/paper/2204.03508/) (2022)
- [Natural Language Processing for Dialects of a Language: A Survey](https://mlsanity.com/paper/2401.05632/) (2024)
- [GlobalWoZ: Globalizing MultiWoZ to Develop Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2110.07679/) (2021)
- [Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation](https://mlsanity.com/paper/2201.13405/) (2022)
- [What Do Dialect Speakers Want? A Survey of Attitudes Towards Language Technology for German Dialects](https://mlsanity.com/paper/2402.11968/) (2024)
- [ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment](https://mlsanity.com/paper/2305.14463/) (2023)
- [A Survey of Corpora for Germanic Low-Resource Languages and Dialects](https://mlsanity.com/paper/2304.09805/) (2023)
- [MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue](https://mlsanity.com/paper/2212.10455/) (2022)
- [Translation and Fusion Improves Zero-shot Cross-lingual Information Extraction](https://mlsanity.com/paper/2305.13582/) (2023)
- [A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialog](https://mlsanity.com/paper/2207.13211/) (2022)
- [Findings of the VarDial Evaluation Campaign 2023](https://mlsanity.com/paper/2305.20080/) (2023)
- [Skill Extraction from Job Postings using Weak Supervision](https://mlsanity.com/paper/2209.08071/) (2022)
- [Meta-Learning a Cross-lingual Manifold for Semantic Parsing](https://mlsanity.com/paper/2209.12577/) (2022)
- [Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languages](https://mlsanity.com/paper/2205.06356/) (2022)
- [Multilingual Auxiliary Tasks Training: Bridging the Gap between Languages for Zero-Shot Transfer of Hate Speech Detection Models](https://mlsanity.com/paper/2210.13029/) (2022)
- [Evaluating Pixel Language Models on Non-Standardized Languages](https://mlsanity.com/paper/2412.09084/) (2024)
- [Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languages](https://mlsanity.com/paper/2303.17683/) (2023)
- [LuxBank: The First Universal Dependency Treebank for Luxembourgish](https://mlsanity.com/paper/2411.04813/) (2024)
- [MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebank](https://mlsanity.com/paper/2403.10293/) (2024)
- [On-Device LLMs for Home Assistant: Dual Role in Intent Detection and Response Generation](https://mlsanity.com/paper/2502.12923/) (2025)
- [BERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Text](https://mlsanity.com/paper/2311.00116/) (2023)
- [Recent Advancements and Challenges of Turkic Central Asian Language Processing](https://mlsanity.com/paper/2407.05006/) (2024)
- [Zero-Shot Slot and Intent Detection in Low-Resource Languages](https://mlsanity.com/paper/2304.13292/) (2023)
- [The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation](https://mlsanity.com/paper/2305.09652/) (2023)
- [Transfer-Free Data-Efficient Multilingual Slot Labeling](https://mlsanity.com/paper/2305.13528/) (2023)
- [Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties](https://mlsanity.com/paper/2402.02078/) (2024)
- [Domain Adaptation in Intent Classification Systems: A Review](https://mlsanity.com/paper/2404.14415/) (2024)
- [Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding](https://mlsanity.com/paper/2205.03656/) (2022)
- [Improving Dialectal Slot and Intent Detection with Auxiliary Tasks: A Multi-Dialectal Bavarian Case Study](https://mlsanity.com/paper/2501.03863/) (2025)
- [Call Larisa Ivanovna: Code-Switching Fools Multilingual NLU Models](https://mlsanity.com/paper/2109.14350/) (2021)

## Abstract (arXiv)

The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce xSID, a new benchmark for cross-lingual Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation transfer works best for intent classification.