mlsanityData to 14 Oct 2026

Multilingual Large Language Models Are Not (Yet) Code-Switchers

arXiv 2305.14235 ↗First posted 23 May 2023cs.CL cs.AIcited by 65

Not on the map. Why?

Abstract (arXiv)

Multilingual Large Language Models (LLMs) have recently shown great capabilities in a wide range of tasks, exhibiting state-of-the-art performance through zero-shot or few-shot prompting methods. While there have been extensive studies on their abilities in monolingual tasks, the investigation of their potential in the context of code-switching (CSW), the practice of alternating languages within an utterance, remains relatively uncharted. In this paper, we provide a comprehensive empirical analysis of various multilingual LLMs, benchmarking their performance across four tasks: sentiment analysis, machine translation, summarization and word-level language identification. Our results indicate that despite multilingual LLMs exhibiting promising outcomes in certain tasks using zero or few-shot prompting, they still underperform in comparison to fine-tuned models of much smaller scales. We argue that current "multilingualism" in LLMs does not inherently imply proficiency with code-switching texts, calling for future research to bridge this discrepancy.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Builds on 47 of its 71 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

47 of 71 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
35 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  8. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  10. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  11. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  12. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  13. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  14. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  15. 2019Massively Multilingual Neural Machine Translationcited by 246
  16. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  17. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  18. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 102
  19. 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 93
  20. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  21. 2022Inverse scaling can become U-shapedcited by 76
  22. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  23. 2020Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Textcited by 67
  24. 2023A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologiescited by 59
  25. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  26. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 57
  27. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  28. 2023Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languagescited by 51
  29. 2020A Sentiment Analysis Dataset for Code-Mixed Malayalam-Englishcited by 50
  30. 2022The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challengescited by 35
  31. 2023Scaling Laws for Multilingual Neural Machine Translationcited by 27
  32. 2022Examining Scaling and Transfer of Language Model Architectures for Machine Translationcited by 26
  33. 2021Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglotscited by 23
  34. 2020Meta-Transfer Learning for Code-Switched Speech Recognitioncited by 22
  35. 2023CroCoSum: A Benchmark Dataset for Cross-Lingual Code-Switched Summarizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 26; 2025: 21; 2026: 12.

65 papers.

Sort:

By area:

  1. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  2. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  3. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  4. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 43
  5. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understandingcited by 34
  6. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  7. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 27
  8. 2025Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causescited by 21
  9. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 19
  10. 2023Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Modelscited by 17
  11. 2023Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparationcited by 16
  12. 2024Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Modelscited by 15
38 more
  1. 2024The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysiscited by 13
  2. 2023Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languagescited by 12
  3. 2024CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Modelscited by 11
  4. 2024MINERS: Multilingual Language Models as Semantic Retrieverscited by 11
  5. 2025PIER: A Novel Metric for Evaluating What Matters in Code-Switchingcited by 11
  6. 2025Aligning LLMs for Multilingual Consistency in Enterprise Applicationscited by 11
  7. 2025Lost in the Mix: Evaluating LLM Understanding of Code-Switched Textcited by 10
  8. 2025MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Modelscited by 8
  9. 2024Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarioscited by 7
  10. 2024Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilitiescited by 7
  11. 2024From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representationcited by 6
  12. 2024Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Modelscited by 6
  13. 2023NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languagescited by 5
  14. 2023InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systemscited by 5
  15. 2024Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switchingcited by 5
  16. 2024From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languagescited by 5
  17. 2024CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedbackcited by 5
  18. 2025Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalitiescited by 5
  19. 2026Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implicationscited by 5
  20. 2024Code-mixed Sentiment and Hate-speech Predictioncited by 4
  21. 2024Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sensecited by 4
  22. 2026The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Modelscited by 4
  23. 2024Multilingual Controlled Generation And Gold-Standard-Agnostic Evaluation of Code-Mixed Sentencescited by 3
  24. 2025Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMscited by 3
  25. 2026OLA: Output Language Alignment in Code-Switched LLM Interactionscited by 3
  26. 2024Revealing the impact of synthetic native samples and multi-tasking strategies in Hindi-English code-mixed humour and sarcasm detectioncited by 2
  27. 2025CMLFormer: A Dual Decoder Transformer with Switching Point Learning for Code-Mixed Language Modelingcited by 2
  28. 2025CS-Sum: A Benchmark for Code-Switching Dialogue Summarization and the Limits of Large Language Modelscited by 2
  29. 2025In-context Language Learning for Endangered Languages in Speech Recognitioncited by 2
  30. 2025MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLPcited by 2
  31. 2025Rethinking Hate Speech Detection on Social Media: Can LLMs Replace Traditional Models?cited by 2
  32. 2025SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMscited by 2
  33. 2023Contextual Code Switching for Machine Translation using Language Modelscited by 1
  34. 2024CoVoSwitch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Unitscited by 1
  35. 2025VietMix: A Naturally-Occurring Parallel Corpus and Augmentation Framework for Vietnamese-English Code-Mixed Machine Translationcited by 1
  36. 2025Adapting Multilingual Models to Code-Mixed Tasks via Model Mergingcited by 1
  37. 2025Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignmentcited by 1
  38. 2024Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations with Effective Data Selectioncited by 0

50 most cited of 65 All 65 (JSON)