# CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP

arXiv: https://arxiv.org/abs/2006.06402 · HTML: https://mlsanity.com/paper/2006.06402/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2020.

## Built on by (most cited first, 30 of 79)

- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [A Survey on Spoken Language Understanding: Recent Advances and New Frontiers](https://mlsanity.com/paper/2103.03095/) (2021)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation](https://mlsanity.com/paper/2112.06223/) (2021)
- [MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER](https://mlsanity.com/paper/2108.13655/) (2021)
- [Consistency Regularization for Cross-Lingual Fine-Tuning](https://mlsanity.com/paper/2106.08226/) (2021)
- [L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models](https://mlsanity.com/paper/2204.08398/) (2022)
- [Understanding Cross-Lingual Alignment -- A Survey](https://mlsanity.com/paper/2404.06228/) (2024)
- [A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding](https://mlsanity.com/paper/2304.04256/) (2023)
- [Enhancing Cross-lingual Transfer by Manifold Mixup](https://mlsanity.com/paper/2205.04182/) (2022)
- [Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Filling](https://mlsanity.com/paper/2103.07792/) (2021)
- [Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training](https://mlsanity.com/paper/2104.08645/) (2021)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation](https://mlsanity.com/paper/2301.12566/) (2023)
- [GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding](https://mlsanity.com/paper/2204.08325/) (2022)
- [Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples](https://mlsanity.com/paper/2111.10962/) (2021)
- [Advances in Multi-turn Dialogue Comprehension: A Survey](https://mlsanity.com/paper/2103.03125/) (2021)
- [Cross-lingual Text Classification with Heterogeneous Graph Neural Network](https://mlsanity.com/paper/2105.11246/) (2021)
- [XeroAlign: Zero-Shot Cross-lingual Transformer Alignment](https://mlsanity.com/paper/2105.02472/) (2021)
- [Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking](https://mlsanity.com/paper/2109.13620/) (2021)
- [Enhancing Cross-lingual Transfer via Phonemic Transcription Integration](https://mlsanity.com/paper/2307.04361/) (2023)
- [Textual Data Augmentation for Arabic-English Code-Switching Speech Recognition](https://mlsanity.com/paper/2201.02550/) (2022)
- [EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching](https://mlsanity.com/paper/2210.12540/) (2022)
- [Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translation](https://mlsanity.com/paper/2303.15265/) (2023)
- [Are the Multilingual Models Better? Improving Czech Sentiment with Transformers](https://mlsanity.com/paper/2108.10640/) (2021)
- [CrossAligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding](https://mlsanity.com/paper/2203.09982/) (2022)

## Abstract (arXiv)

Multi-lingual contextualized embeddings, such as multilingual-BERT (mBERT), have shown success in a variety of zero-shot cross-lingual tasks. However, these models are limited by having inconsistent contextualized representations of subwords across different languages. Existing work addresses this issue by bilingual projection and fine-tuning technique. We propose a data augmentation framework to generate multi-lingual code-switching data to fine-tune mBERT, which encourages model to align representations from source and multiple target languages once by mixing their context information. Compared with the existing work, our method does not rely on bilingual sentences for training, and requires only one training process for multiple target languages. Experimental results on five tasks with 19 languages show that our method leads to significantly improved performances for all the tasks compared with mBERT.