# Multilingual Large Language Models Are Not (Yet) Code-Switchers

arXiv: https://arxiv.org/abs/2305.14235 · HTML: https://mlsanity.com/paper/2305.14235/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 71 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 47)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Unified Scaling Laws for Routed Language Models](https://mlsanity.com/paper/2202.01169/) (2022)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [Scaling Laws for Generative Mixed-Modal Language Models](https://mlsanity.com/paper/2301.03728/) (2023)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)

## Built on by (most cited first, 30 of 65)

- [Understanding and Mitigating Language Confusion in LLMs](https://mlsanity.com/paper/2406.20052/) (2024)
- [LLMs Are Few-Shot In-Context Low-Resource Language Learners](https://mlsanity.com/paper/2403.16512/) (2024)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages](https://mlsanity.com/paper/2406.10118/) (2024)
- [Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding](https://mlsanity.com/paper/2406.15481/) (2024)
- [Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ](https://mlsanity.com/paper/2403.03814/) (2024)
- [Code-Switching Curriculum Learning for Multilingual Transfer in LLMs](https://mlsanity.com/paper/2411.02460/) (2024)
- [Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes](https://mlsanity.com/paper/2505.14815/) (2025)
- [Evaluating Morphological Compositional Generalization in Large Language Models](https://mlsanity.com/paper/2410.12656/) (2024)
- [Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models](https://mlsanity.com/paper/2309.05454/) (2023)
- [Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation](https://mlsanity.com/paper/2310.20470/) (2023)
- [Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models](https://mlsanity.com/paper/2402.14714/) (2024)
- [The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis](https://mlsanity.com/paper/2402.12976/) (2024)
- [Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languages](https://mlsanity.com/paper/2310.03018/) (2023)
- [CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models](https://mlsanity.com/paper/2402.13109/) (2024)
- [MINERS: Multilingual Language Models as Semantic Retrievers](https://mlsanity.com/paper/2406.07424/) (2024)
- [PIER: A Novel Metric for Evaluating What Matters in Code-Switching](https://mlsanity.com/paper/2501.09512/) (2025)
- [Aligning LLMs for Multilingual Consistency in Enterprise Applications](https://mlsanity.com/paper/2509.23659/) (2025)
- [Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text](https://mlsanity.com/paper/2506.14012/) (2025)
- [MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models](https://mlsanity.com/paper/2507.20395/) (2025)
- [Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios](https://mlsanity.com/paper/2406.00343/) (2024)
- [Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities](https://mlsanity.com/paper/2410.11079/) (2024)
- [From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation](https://mlsanity.com/paper/2404.09138/) (2024)
- [Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Models](https://mlsanity.com/paper/2410.22660/) (2024)
- [NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages](https://mlsanity.com/paper/2309.10661/) (2023)
- [InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2310.08885/) (2023)
- [Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switching](https://mlsanity.com/paper/2410.18436/) (2024)
- [From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages](https://mlsanity.com/paper/2410.18836/) (2024)
- [CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedback](https://mlsanity.com/paper/2411.09073/) (2024)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)

## Abstract (arXiv)

Multilingual Large Language Models (LLMs) have recently shown great capabilities in a wide range of tasks, exhibiting state-of-the-art performance through zero-shot or few-shot prompting methods. While there have been extensive studies on their abilities in monolingual tasks, the investigation of their potential in the context of code-switching (CSW), the practice of alternating languages within an utterance, remains relatively uncharted. In this paper, we provide a comprehensive empirical analysis of various multilingual LLMs, benchmarking their performance across four tasks: sentiment analysis, machine translation, summarization and word-level language identification. Our results indicate that despite multilingual LLMs exhibiting promising outcomes in certain tasks using zero or few-shot prompting, they still underperform in comparison to fine-tuned models of much smaller scales. We argue that current "multilingualism" in LLMs does not inherently imply proficiency with code-switching texts, calling for future research to bridge this discrepancy.