# Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text

arXiv: https://arxiv.org/abs/2506.14012 · HTML: https://mlsanity.com/paper/2506.14012/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://mlsanity.com/paper/2008.04277/) (2020)
- [A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies](https://mlsanity.com/paper/2301.01967/) (2023)
- [Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences](https://mlsanity.com/paper/1909.08582/) (2019)
- [ALLaM: Large Language Models for Arabic and English](https://mlsanity.com/paper/2407.15390/) (2024)
- [Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systems](https://mlsanity.com/paper/1911.09273/) (2019)
- [Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots](https://mlsanity.com/paper/2103.09593/) (2021)
- [LLM as a Broken Telephone: Iterative Generation Distorts Information](https://mlsanity.com/paper/2502.20258/) (2025)
- [From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences](https://mlsanity.com/paper/2405.05572/) (2024)
- [Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities](https://mlsanity.com/paper/2410.11079/) (2024)
- [LLM-based Code-Switched Text Generation for Grammatical Error Correction](https://mlsanity.com/paper/2410.10349/) (2024)
- [What talking you?: Translating Code-Mixed Messaging Texts to English](https://mlsanity.com/paper/2411.05253/) (2024)
- [AdvCodeMix: Adversarial Attack on Code-Mixed Data](https://mlsanity.com/paper/2111.00350/) (2021)
- [Conditioning LLMs to Generate Code-Switched Text](https://mlsanity.com/paper/2502.12924/) (2025)
- [The Impact of Syntactic and Semantic Proximity on Machine Translation with Back-Translation](https://mlsanity.com/paper/2403.18031/) (2024)
- [Grammatical Error Correction for Code-Switched Sentences by Learners of English](https://mlsanity.com/paper/2404.12489/) (2024)

## Built on by (most cited first, 10 of 10)

- [Culture is Everywhere: A Call for Intentionally Cultural Evaluation](https://mlsanity.com/paper/2509.01301/) (2025)
- [On the Entity-Level Alignment in Crosslingual Consistency](https://mlsanity.com/paper/2510.10280/) (2025)
- [Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMs](https://mlsanity.com/paper/2510.05678/) (2025)
- [Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment](https://mlsanity.com/paper/2511.10670/) (2025)
- [Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communities](https://mlsanity.com/paper/2605.04127/) (2026)
- [Code-Switching Reveals Language Anchoring in Multilingual LLMs](https://mlsanity.com/paper/2606.19668/) (2026)
- [Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO](https://mlsanity.com/paper/2510.01616/) (2025)
- [Code Mixologist : A Practitioner's Guide to Building Code-Mixed LLMs](https://mlsanity.com/paper/2602.11181/) (2026)
- [BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Models](https://mlsanity.com/paper/2608.27268/) (2026)
- [CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing](https://mlsanity.com/paper/2610.20272/) (2026)

## Abstract (arXiv)

Code-switching (CSW) is the act of alternating between two or more languages within a single discourse. This phenomenon is widespread in multilingual communities, and increasingly prevalent in online content, where users naturally mix languages in everyday communication. As a result, Large Language Models (LLMs), now central to content processing and generation, are frequently exposed to code-switched inputs. Given their widespread use, it is crucial to understand how LLMs process and reason about such mixed-language text. This paper presents a systematic evaluation of LLM comprehension under code-switching by generating CSW variants of established reasoning and comprehension benchmarks. While degradation is evident when foreign tokens disrupt English textunicodex2013even under linguistic constraintsunicodex2013embedding English into other languages often improves comprehension. Though prompting yields mixed results, fine-tuning offers a more stable path to degradation mitigation.