mlsanityData to 14 Oct 2026

Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training

arXiv 2504.01801 ↗First posted 2 Apr 2025cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) exhibit remarkable multilingual capabilities despite the extreme language imbalance in the pre-training data. In this paper, we closely examine the reasons behind this phenomenon, focusing on the pre-training corpus. We find that the existence of code-switching, alternating between different languages within a context, is key to multilingual capabilities. We conduct an analysis to investigate code-switching in the pre-training corpus, examining its presence and categorizing it into four types within two quadrants. We then assess its impact on multilingual performance. These types of code-switching data are unbalanced in proportions and demonstrate different effects on facilitating language transfer. To better explore the power of code-switching for language alignment during pre-training, we investigate the strategy of synthetic code-switching. We continuously scale up the synthetic code-switching data and observe remarkable improvements in both benchmarks and representation space. Extensive experiments indicate that incorporating synthetic code-switching data enables better language alignment and generalizes well to high, medium, and low-resource languages with pre-training corpora of varying qualities.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 35 of its 56 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

35 of 56 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
23 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  2. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  3. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  4. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  5. 2019How multilingual is Multilingual BERT?cited by 645
  6. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  7. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  8. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  9. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  10. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  11. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  12. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  13. 2020OCNLI: Original Chinese Natural Language Inferencecited by 86
  14. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimizationcited by 69
  15. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 65
  16. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  17. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  18. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understandingcited by 34
  19. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 27
  20. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  21. 2025OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Trainingcited by 21
  22. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  23. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025The Impact of Language Mixing on Bilingual LLM Reasoningcited by 12
  2. 2025Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Modelscited by 11
  3. 2025Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalitiescited by 5
  4. 2026Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Modelscited by 5
  5. 2025Trillion 7B Technical Reportcited by 4
  6. 2025Controlling Language Confusion in Multilingual LLMscited by 3
  7. 2025Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMscited by 3
  8. 2025SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMscited by 2
  9. 2026Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuningcited by 2
  10. 2026The Role of Mixed-Language Documents for Multilingual Large Language Model Pretrainingcited by 1
  11. 2026PolyFact: Comparing Consistency-Driven Post-training Methods for Cross-Lingual Factual Recallcited by 1
  12. 2025Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPOcited by 0
4 more
  1. 2026A Study of Crosslinguistic Influence in Language Modelscited by 0
  2. 2026Multilingual Knowledge Transfer under Data Constraints via Lexical Interventionscited by 0
  3. 2026Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergencecited by 0
  4. 2026Introducing Code-Switched Contexts to Cognitively-Inspired Bilingual Model Trainingcited by 0