mlsanityData to 14 Oct 2026

Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text

arXiv 2506.14012 ↗First posted 16 Jun 2025cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Code-switching (CSW) is the act of alternating between two or more languages within a single discourse. This phenomenon is widespread in multilingual communities, and increasingly prevalent in online content, where users naturally mix languages in everyday communication. As a result, Large Language Models (LLMs), now central to content processing and generation, are frequently exposed to code-switched inputs. Given their widespread use, it is crucial to understand how LLMs process and reason about such mixed-language text. This paper presents a systematic evaluation of LLM comprehension under code-switching by generating CSW variants of established reasoning and comprehension benchmarks. While degradation is evident when foreign tokens disrupt English textunicodex2013even under linguistic constraintsunicodex2013embedding English into other languages often improves comprehension. Though prompting yields mixed results, fine-tuning offers a more stable path to degradation mitigation.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

30 of 52 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  9. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  10. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  11. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  12. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 126
18 more
  1. 2020LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluationcited by 93
  2. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  3. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  4. 2020SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweetscited by 62
  5. 2023A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologiescited by 59
  6. 2019Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentencescited by 55
  7. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  8. 2019Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systemscited by 42
  9. 2021Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglotscited by 23
  10. 2025LLM as a Broken Telephone: Iterative Generation Distorts Informationcited by 12
  11. 2024From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentencescited by 8
  12. 2024Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilitiescited by 7
  13. 2024LLM-based Code-Switched Text Generation for Grammatical Error Correctioncited by 4
  14. 2024What talking you?: Translating Code-Mixed Messaging Texts to Englishcited by 4
  15. 2021AdvCodeMix: Adversarial Attack on Code-Mixed Datacited by 3
  16. 2025Conditioning LLMs to Generate Code-Switched Textcited by 3
  17. 2024The Impact of Syntactic and Semantic Proximity on Machine Translation with Back-Translationcited by 2
  18. 2024Grammatical Error Correction for Code-Switched Sentences by Learners of Englishcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Culture is Everywhere: A Call for Intentionally Cultural Evaluationcited by 10
  2. 2025On the Entity-Level Alignment in Crosslingual Consistencycited by 4
  3. 2025Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMscited by 3
  4. 2025Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignmentcited by 1
  5. 2026Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communitiescited by 1
  6. 2026Code-Switching Reveals Language Anchoring in Multilingual LLMscited by 1
  7. 2025Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPOcited by 0
  8. 2026Code Mixologist : A Practitioner's Guide to Building Code-Mixed LLMscited by 0
  9. 2026BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Modelscited by 0
  10. 2026CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writingcited by 0