# Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languages

arXiv: https://arxiv.org/abs/2303.13592 · HTML: https://mlsanity.com/paper/2303.13592/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 94 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Does Synthetic Data Generation of LLMs Help Clinical Text Mining?](https://mlsanity.com/paper/2303.04360/) (2023)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages](https://mlsanity.com/paper/2205.15960/) (2022)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Data Feedback Loops: Model-driven Amplification of Dataset Biases](https://mlsanity.com/paper/2209.03942/) (2022)
- [A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies](https://mlsanity.com/paper/2301.01967/) (2023)
- [Are Multilingual Models Effective in Code-Switching?](https://mlsanity.com/paper/2103.13309/) (2021)
- [Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences](https://mlsanity.com/paper/1909.08582/) (2019)
- [ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation](https://mlsanity.com/paper/2112.06223/) (2021)
- [Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systems](https://mlsanity.com/paper/1911.09273/) (2019)
- [NusaCrowd: Open Source Initiative for Indonesian NLP Resources](https://mlsanity.com/paper/2212.09648/) (2022)
- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Text](https://mlsanity.com/paper/2107.06483/) (2021)
- [Learning Fast Adaptation on Cross-Accented Speech Recognition](https://mlsanity.com/paper/2003.01901/) (2020)
- [M2D2: A Massively Multi-domain Language Modeling Dataset](https://mlsanity.com/paper/2210.07370/) (2022)
- [Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots](https://mlsanity.com/paper/2103.09593/) (2021)
- [On Language Models for Creoles](https://mlsanity.com/paper/2109.06074/) (2021)
- [Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing](https://mlsanity.com/paper/2105.08807/) (2021)
- [CroCoSum: A Benchmark Dataset for Cross-Lingual Code-Switched Summarization](https://mlsanity.com/paper/2303.04092/) (2023)

## Built on by (most cited first, 30 of 51)

- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [Understanding and Mitigating Language Confusion in LLMs](https://mlsanity.com/paper/2406.20052/) (2024)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [LLMs Are Few-Shot In-Context Low-Resource Language Learners](https://mlsanity.com/paper/2403.16512/) (2024)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination](https://mlsanity.com/paper/2406.08818/) (2024)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages](https://mlsanity.com/paper/2406.10118/) (2024)
- [Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding](https://mlsanity.com/paper/2406.15481/) (2024)
- [Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ](https://mlsanity.com/paper/2403.03814/) (2024)
- [Code-Switching Curriculum Learning for Multilingual Transfer in LLMs](https://mlsanity.com/paper/2411.02460/) (2024)
- [InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning](https://mlsanity.com/paper/2305.13627/) (2023)
- [Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes](https://mlsanity.com/paper/2505.14815/) (2025)
- [MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English Clinical Queries](https://mlsanity.com/paper/2401.01596/) (2024)
- [How Vocabulary Sharing Facilitates Multilingualism in LLaMA?](https://mlsanity.com/paper/2311.09071/) (2023)
- [The Multilingual Divide and Its Impact on Global AI Safety](https://mlsanity.com/paper/2505.21344/) (2025)
- [CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages](https://mlsanity.com/paper/2507.18791/) (2025)
- [Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia](https://mlsanity.com/paper/2503.07920/) (2025)
- [Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities](https://mlsanity.com/paper/2410.11079/) (2024)
- [Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Models](https://mlsanity.com/paper/2410.22660/) (2024)
- [It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs](https://mlsanity.com/paper/2508.19089/) (2025)
- [NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages](https://mlsanity.com/paper/2309.10661/) (2023)
- [InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2310.08885/) (2023)
- [Standard Language Ideology in AI-Generated Language](https://mlsanity.com/paper/2406.08726/) (2024)
- [Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switching](https://mlsanity.com/paper/2410.18436/) (2024)
- [A Survey of Code-switched Arabic NLP: Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2501.13419/) (2025)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sense](https://mlsanity.com/paper/2410.21573/) (2024)
- [Leveraging Large Language Models for Code-Mixed Data Augmentation in Sentiment Analysis](https://mlsanity.com/paper/2411.00691/) (2024)

## Abstract (arXiv)

While code-mixing is a common linguistic practice in many parts of the world, collecting high-quality and low-cost code-mixed data remains a challenge for natural language processing (NLP) research. The recent proliferation of Large Language Models (LLMs) compels one to ask: how capable are these systems in generating code-mixed data? In this paper, we explore prompting multilingual LLMs in a zero-shot manner to generate code-mixed data for seven languages in South East Asia (SEA), namely Indonesian, Malay, Chinese, Tagalog, Vietnamese, Tamil, and Singlish. We find that publicly available multilingual instruction-tuned models such as BLOOMZ and Flan-T5-XXL are incapable of producing texts with phrases or clauses from different languages. ChatGPT exhibits inconsistent capabilities in generating code-mixed texts, wherein its performance varies depending on the prompt template and language pairing. For instance, ChatGPT generates fluent and natural Singlish texts (an English-based creole spoken in Singapore), but for English-Tamil language pair, the system mostly produces grammatically incorrect or semantically meaningless utterances. Furthermore, it may erroneously introduce languages not specified in the prompt. Based on our investigation, existing multilingual LLMs exhibit a wide range of proficiency in code-mixed data generation for SEA languages. As such, we advise against using LLMs in this context without extensive human checks.