Call Larisa Ivanovna: Code-Switching Fools Multilingual NLU Models
Not on the map. Why?
Abstract (arXiv)
Practical needs of developing task-oriented dialogue assistants require the ability to understand many languages. Novel benchmarks for multilingual natural language understanding (NLU) include monolingual sentences in several languages, annotated with intents and slots. In such setup models for cross-lingual transfer show remarkable performance in joint intent recognition and slot filling. However, existing benchmarks lack of code-switched utterances, which are difficult to gather and label due to complexity in the grammatical structure. The evaluation of NLU models seems biased and limited, since code-switching is being left out of scope. Our work adopts recognized methods to generate plausible and naturally-sounding code-switched utterances and uses them to create a synthetic code-switched test set. Based on experiments, we report that the state-of-the-art NLU models are unable to handle code-switching. At worst, the performance, evaluated by semantic accuracy, drops as low as 15% from 80% across languages. Further we show, that pre-training on synthetic code-mixed data helps to maintain performance on the proposed test set at a comparable level with monolingual data. Finally, we analyze different language pairs and show that the closer the languages are, the better the NLU model handles their alternation. This is in line with the common understanding of how multilingual models conduct transferring between languages
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 29 Sep 2021.
Builds on
33 of 47 references.
References · 33
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
- 2020Beyond English-Centric Multilingual Machine Translationcited by 561
- 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
- 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
- 2019BERT for Joint Intent Classification and Slot Fillingcited by 196
- 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
- 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
- 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
- 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 102
21 more
- 2018Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examplescited by 94
- 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 93
- 2020End-to-End Slot Alignment and Recognition for Cross-Lingual NLUcited by 83
- 2020CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLPcited by 79
- 2021Are Multilingual Models Effective in Code-Switching?cited by 56
- 2019Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentencescited by 55
- 2021From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understandingcited by 43
- 2019Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systemscited by 42
- 2019XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answeringcited by 35
- 2021Grey-box Adversarial Attack And Defence For Sentiment Classificationcited by 28
- 2021Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Fillingcited by 25
- 2021Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglotscited by 23
- 2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 17
- 2021Translate & Fill: Improving Zero-Shot Multilingual Semantic Parsing with Synthetic Datacited by 17
- 2021Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixingcited by 16
- 2019A Deep Generative Model for Code-Switched Textcited by 13
- 2018Acoustic and Textual Data Augmentation for Improved ASR of Code-Switching Speechcited by 12
- 2021XeroAlign: Zero-Shot Cross-lingual Transformer Alignmentcited by 11
- 2021El Volumen Louder Por Favor: Code-switching in Task-oriented Semantic Parsingcited by 8
- 2020Cross-lingual Spoken Language Understanding with Regularized Representation Alignmentcited by 5
- 2021Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understandingcited by 4
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.