mlsanityData to 7 Oct 2026

Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources

arXiv 2504.04152 ↗First posted 5 Apr 2025cs.CLcited by 10

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) exhibit significant disparities in performance across languages, primarily benefiting high-resource languages while marginalizing underrepresented ones. Continual Pretraining (CPT) has emerged as a promising approach to address this imbalance, although the relative effectiveness of monolingual, bilingual, and code-augmented data strategies remains unclear. This study systematically evaluates 36 CPT configurations involving three multilingual base models, across 30+ languages categorized as altruistic, selfish, and stagnant, spanning various resource levels. Our findings reveal three major insights: (1) Bilingual CPT improves multilingual classification but often causes language mixing issues during generation. (2) Including programming code data during CPT consistently enhances multilingual classification accuracy, particularly benefiting low-resource languages, but introduces a trade-off by slightly degrading generation quality. (3) Contrary to prior work, we observe substantial deviations from language classifications according to their impact on cross-lingual transfer: Languages classified as altruistic often negatively affect related languages, selfish languages show conditional and configuration-dependent behavior, and stagnant languages demonstrate surprising adaptability under certain CPT conditions. These nuanced interactions emphasize the complexity of multilingual representation learning, underscoring the importance of systematic studies on generalizable language classification to inform future multilingual CPT strategies.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 42 of its 58 references that are in the map. First posted on arXiv 5 Apr 2025.

Builds on

42 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?28 Sep
  2. 2022Language Models of Code are Few-Shot Commonsense Learners13 Oct

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  10. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  11. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  12. 2022The Stack: 3 TB of permissively licensed source codecited by 354
28 more
  1. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  2. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  3. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  4. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  5. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  6. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  7. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  8. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  9. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  10. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  11. 2022Code Translation with Compiler Representationscited by 42
  12. 2023GlotLID: Language Identification for Low-Resource Languagescited by 37
  13. 2023Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrationscited by 34
  14. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  15. 2024Code Pretraining Improves Entity Tracking Abilities of Language Modelscited by 31
  16. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  17. 2025Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Trainingcited by 23
  18. 2021nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?cited by 21
  19. 2022Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translationcited by 21
  20. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 21
  21. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  22. 2025Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakerscited by 16
  23. 2024Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scalecited by 15
  24. 2023How Vocabulary Sharing Facilitates Multilingualism in LLaMA?cited by 12
  25. 2024RedWhale: An Adapted Korean LLM Through Efficient Continual Pretrainingcited by 8
  26. 2024Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning?cited by 6
  27. 2024The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Mergingcited by 3
  28. 2024Ayn: A Tiny yet Competitive Indian Legal Language Model Pretrained from Scratchcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19
  2. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  3. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  4. 2026Soro: A Lightweight Foundation Model and Chatbot for Tajikcited by 2
  5. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  6. 2025Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Modelscited by 0
  7. 2026Diffutron: A Masked Diffusion Language Model for Turkish Languagecited by 0
  8. 2026SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignmentcited by 0
  9. 2026MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languagescited by 0
  10. 2026From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024What Drives Performance in Multilingual Language Models?
  2. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  3. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning
  4. 2025A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics
  5. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts
  6. 2024Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation
  7. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context
  8. 2024Beyond Data Quantity: Key Factors Driving Performance in Multilingual Language Models