mlsanityData to 14 Oct 2026

From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

arXiv 2105.07316 ↗First posted 15 May 2021cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce xSID, a new benchmark for cross-lingual Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation transfer works best for intent classification.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 15 May 2021.

Builds on

33 of 56 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  12. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
21 more
  1. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  2. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  3. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  4. 2019BERT for Joint Intent Classification and Slot Fillingcited by 196
  5. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  6. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  7. 2018Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialogcited by 125
  8. 2017Offline bilingual word vectors, orthogonal transformations and the inverted softmaxcited by 96
  9. 2020End-to-End Slot Alignment and Recognition for Cross-Lingual NLUcited by 83
  10. 2016A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Taskscited by 62
  11. 2020Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLPcited by 50
  12. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  13. 2019Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Modelcited by 24
  14. 2019On the use of BERT for Neural Machine Translationcited by 23
  15. 2019Zero-shot Cross-lingual Dialogue Systems with Transferable Latent Variablescited by 21
  16. 2019Almawave-SLU: A new dataset for SLU in Italiancited by 13
  17. 2021El Volumen Louder Por Favor: Code-switching in Task-oriented Semantic Parsingcited by 8
  18. 2020Don't Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddingscited by 5
  19. 2020Multi-Domain Spoken Language Understanding Using Domain- and Task-Aware Parameterizationcited by 4
  20. 2016Deep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchiescited by 1
  21. 2018Selecting Machine-Translated Data for Quick Bootstrapping of a Natural Language Understanding Systemcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 10; 2024: 9; 2025: 7; 2026: 5.

43 papers.

Sort:

By area:

  1. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  2. 2024Natural Language Processing for Dialects of a Language: A Surveycited by 40
  3. 2021GlobalWoZ: Globalizing MultiWoZ to Develop Multilingual Task-Oriented Dialogue Systemscited by 31
  4. 2022Cross-Lingual Dialogue Dataset Creation via Outline-Based Generationcited by 23
  5. 2024What Do Dialect Speakers Want? A Survey of Attitudes Towards Language Technology for German Dialectscited by 21
  6. 2023ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessmentcited by 19
  7. 2023A Survey of Corpora for Germanic Low-Resource Languages and Dialectscited by 16
  8. 2022MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialoguecited by 15
  9. 2023Translation and Fusion Improves Zero-shot Cross-lingual Information Extractioncited by 15
  10. 2022A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialogcited by 14
  11. 2023Findings of the VarDial Evaluation Campaign 2023cited by 14
  12. 2022Skill Extraction from Job Postings using Weak Supervisioncited by 13
31 more
  1. 2022Meta-Learning a Cross-lingual Manifold for Semantic Parsingcited by 11
  2. 2022Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languagescited by 10
  3. 2022Multilingual Auxiliary Tasks Training: Bridging the Gap between Languages for Zero-Shot Transfer of Hate Speech Detection Modelscited by 10
  4. 2024Evaluating Pixel Language Models on Non-Standardized Languagescited by 10
  5. 2023Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languagescited by 9
  6. 2024LuxBank: The First Universal Dependency Treebank for Luxembourgishcited by 9
  7. 2024MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebankcited by 7
  8. 2025On-Device LLMs for Home Assistant: Dual Role in Intent Detection and Response Generationcited by 7
  9. 2023BERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Textcited by 6
  10. 2024Recent Advancements and Challenges of Turkic Central Asian Language Processingcited by 6
  11. 2023Zero-Shot Slot and Intent Detection in Low-Resource Languagescited by 4
  12. 2023The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translationcited by 4
  13. 2023Transfer-Free Data-Efficient Multilingual Slot Labelingcited by 4
  14. 2024Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varietiescited by 4
  15. 2024Domain Adaptation in Intent Classification Systems: A Reviewcited by 4
  16. 2022Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understandingcited by 3
  17. 2025Improving Dialectal Slot and Intent Detection with Auxiliary Tasks: A Multi-Dialectal Bavarian Case Studycited by 3
  18. 2021Call Larisa Ivanovna: Code-Switching Fools Multilingual NLU Modelscited by 2
  19. 2023Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsingcited by 2
  20. 2025Add Noise, Tasks, or Layers? MaiNLP at the VarDial 2025 Shared Task on Norwegian Dialectal Slot and Intent Detectioncited by 2
  21. 2025The Devil Is in the Word Alignment Details: On Translation-Based Cross-Lingual Transfer for Token Classification Taskscited by 2
  22. 2025Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialectscited by 2
  23. 2022Language Varieties of Italy: Technology Challenges and Opportunitiescited by 1
  24. 2024HiTZ at VarDial 2025 NorSID: Overcoming Data Scarcity with Language Transfer and Automatic Data Annotationcited by 1
  25. 2025Multimodal Evaluation of Russian-language Architecturescited by 1
  26. 2026Indirect Question Answering in English, German and Bavarian: A Challenging Task for High- and Low-Resource Languages Alikecited by 1
  27. 2026IntentGrasp: A Comprehensive Benchmark for Intent Understandingcited by 1
  28. 2025TransAlign: Machine Translation Encoders are Strong Word Aligners, Toocited by 0
  29. 2026textitVersteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Languagecited by 0
  30. 2026ltzGLUE: Luxembourgish General Language Understanding Evaluationcited by 0
  31. 2026Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustnesscited by 0