# Are Multilingual Models Effective in Code-Switching?

arXiv: https://arxiv.org/abs/2103.13309 · HTML: https://mlsanity.com/paper/2103.13309/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding](https://mlsanity.com/paper/2009.05387/) (2020)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion](https://mlsanity.com/paper/1804.07745/) (2018)
- [Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences](https://mlsanity.com/paper/1909.08582/) (2019)
- [BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages](https://mlsanity.com/paper/1710.02187/) (2017)
- [Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systems](https://mlsanity.com/paper/1911.09273/) (2019)
- [XPersona: Evaluating Multilingual Personalized Chatbot](https://mlsanity.com/paper/2003.07568/) (2020)
- [Named Entity Recognition on Code-Switched Data: Overview of the CALCS 2018 Shared Task](https://mlsanity.com/paper/1906.04138/) (2019)
- [Meta-Transfer Learning for Code-Switched Speech Recognition](https://mlsanity.com/paper/2004.14228/) (2020)
- [Multilingual Models for Compositional Distributed Semantics](https://mlsanity.com/paper/1404.4641/) (2014)
- [A Multi-task Approach for Named Entity Recognition in Social Media Data](https://mlsanity.com/paper/1906.04135/) (2019)
- [Frustratingly Easy Meta-Embedding -- Computing Meta-Embeddings by Averaging Source Word Embeddings](https://mlsanity.com/paper/1804.05262/) (2018)
- [Dynamic Meta-Embeddings for Improved Sentence Representations](https://mlsanity.com/paper/1804.07983/) (2018)
- [Code-Switching Language Modeling using Syntax-Aware Multi-Task Learning](https://mlsanity.com/paper/1805.12070/) (2018)
- [A Deep Generative Model for Code-Switched Text](https://mlsanity.com/paper/1906.08972/) (2019)
- [Learn to Code-Switch: Data Augmentation using Copy Mechanism on Language Modeling](https://mlsanity.com/paper/1810.10254/) (2018)
- [Think Globally, Embed Locally --- Locally Linear Meta-embedding of Words](https://mlsanity.com/paper/1709.06671/) (2017)
- [Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition](https://mlsanity.com/paper/1909.08504/) (2019)
- [Linear Ensembles of Word Embedding Models](https://mlsanity.com/paper/1704.01419/) (2017)

## Built on by (most cited first, 30 of 56)

- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [LLMs Are Few-Shot In-Context Low-Resource Language Learners](https://mlsanity.com/paper/2403.16512/) (2024)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation](https://mlsanity.com/paper/2104.08200/) (2021)
- [Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languages](https://mlsanity.com/paper/2303.13592/) (2023)
- [ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation](https://mlsanity.com/paper/2112.06223/) (2021)
- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2105.03953/) (2021)
- [End-to-End Speech Translation for Code Switched Speech](https://mlsanity.com/paper/2204.05076/) (2022)
- [RoBERTuito: a pre-trained language model for social media text in Spanish](https://mlsanity.com/paper/2111.09453/) (2021)
- [Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languages](https://mlsanity.com/paper/2310.03018/) (2023)
- [MINERS: Multilingual Language Models as Semantic Retrievers](https://mlsanity.com/paper/2406.07424/) (2024)
- [Code-switched inspired losses for generic spoken dialog representations](https://mlsanity.com/paper/2108.12465/) (2021)
- [Yes but.. Can ChatGPT Identify Entities in Historical Documents?](https://mlsanity.com/paper/2303.17322/) (2023)
- [CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages](https://mlsanity.com/paper/2507.18791/) (2025)
- [CodemixedNLP: An Extensible and Open NLP Toolkit for Code-Mixing](https://mlsanity.com/paper/2106.06004/) (2021)
- [CroCoSum: A Benchmark Dataset for Cross-Lingual Code-Switched Summarization](https://mlsanity.com/paper/2303.04092/) (2023)
- [Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text](https://mlsanity.com/paper/2108.01861/) (2021)
- [Speaker Information Can Guide Models to Better Inductive Biases: A Case Study On Predicting Code-Switching](https://mlsanity.com/paper/2203.08979/) (2022)
- [m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt](https://mlsanity.com/paper/2403.17556/) (2024)
- [From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation](https://mlsanity.com/paper/2404.09138/) (2024)
- [Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Models](https://mlsanity.com/paper/2410.22660/) (2024)
- [Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation](https://mlsanity.com/paper/2108.10808/) (2021)
- [Switch Point biased Self-Training: Re-purposing Pretrained Models for Code-Switching](https://mlsanity.com/paper/2111.01231/) (2021)
- [SNP2Vec: Scalable Self-Supervised Pre-Training for Genome-Wide Association Study](https://mlsanity.com/paper/2204.06699/) (2022)
- [Every picture tells a story: Image-grounded controllable stylistic story generation](https://mlsanity.com/paper/2209.01638/) (2022)
- [IndoRobusta: Towards Robustness Against Diverse Code-Mixed Indonesian Local Languages](https://mlsanity.com/paper/2311.12405/) (2023)
- [From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages](https://mlsanity.com/paper/2410.18836/) (2024)

## Abstract (arXiv)

Multilingual language models have shown decent performance in multilingual and cross-lingual natural language understanding tasks. However, the power of these multilingual models in code-switching tasks has not been fully explored. In this paper, we study the effectiveness of multilingual language models to understand their capability and adaptability to the mixed-language setting by considering the inference speed, performance, and number of parameters to measure their practicality. We conduct experiments in three language pairs on named entity recognition and part-of-speech tagging and compare them with existing methods, such as using bilingual embeddings and multilingual meta-embeddings. Our findings suggest that pre-trained multilingual models do not necessarily guarantee high-quality representations on code-switching, while using meta-embeddings achieves similar results with significantly fewer parameters.