mlsanityData to 14 Oct 2026

Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data

arXiv 2506.00469 ↗First posted 31 May 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper investigates a critical design decision in the practice of massively multilingual continual pre-training -- the inclusion of parallel data. Specifically, we study the impact of bilingual translation data for massively multilingual language adaptation of the Llama3 family of models to 500 languages. To this end, we construct the MaLA bilingual translation corpus, containing data from more than 2,500 language pairs. Subsequently, we develop the EMMA-500 Llama 3 suite of four massively multilingual models -- continually pre-trained from the Llama 3 family of base models extensively on diverse data mixes up to 671B tokens -- and explore the effect of continual pre-training with or without bilingual translation data. Comprehensive evaluation across 7 tasks and 12 benchmarks demonstrates that bilingual data tends to enhance language transfer and performance, particularly for low-resource languages. We open-source the MaLA corpus, EMMA-500 Llama 3 suite artefacts, code, and model generations.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 70 of its 94 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

70 of 94 references.

References · 70

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
38 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  2. 2024Qwen2 Technical Reportcited by 2,120
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  9. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  10. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  11. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  12. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  13. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  14. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  15. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  16. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  17. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  18. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  19. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 200
  20. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  21. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  22. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  23. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  24. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  25. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  26. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 128
  27. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  28. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  29. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 98
  30. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
  31. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  32. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  33. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  34. 2025Overtrained Language Models Are Harder to Fine-Tunecited by 58
  35. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 53
  36. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  37. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 48
  38. 2022CSL: A Large-scale Chinese Scientific Literature Datasetcited by 45

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  2. 2026M3Kang: Evaluating Multilingual Multimodal Mathematical Reasoning in Vision-Language Modelscited by 2
  3. 2025Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languagescited by 0
  4. 2026LatentMT: Machine Translation with Latent Reasoningcited by 0