mlsanityData to 7 Oct 2026

EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

arXiv 2409.17892 ↗First posted 26 Sep 2024cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability. We release the MaLA corpus, EMMA-500 model weights, scripts, and model generations.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 73 of its 131 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

73 of 131 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
38 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  8. 2022Galactica: A Large Language Model for Sciencecited by 736
  9. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  12. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  13. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  14. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  15. 2016FastText.zip: Compressing text classification modelscited by 331
  16. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  17. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  18. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  19. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  20. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  21. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  22. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  23. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  24. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  25. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  26. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  27. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  28. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  29. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  30. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  31. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 127
  32. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 121
  33. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  34. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 97
  35. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  36. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  37. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  38. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 74

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  2. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  3. 2025Scaling Low-Resource MT via Synthetic Data Generation with LLMscited by 14
  4. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 14
  5. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  6. 2025How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspectivecited by 8
  7. 2025DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detectioncited by 5
  8. 2025The Art of Asking: Multilingual Prompt Optimization for Synthetic Datacited by 5
  9. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  10. 2025Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Modelcited by 4
  11. 2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
  12. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
4 more
  1. 2025Automatic Machine Translation Detection Using a Surrogate Multilingual Translation Modelcited by 0
  2. 2025MiniLingua: A Small Open-Source LLM for European Languagescited by 0
  3. 2026VEPO: Variable Entropy Policy Optimization for Low-Resource Language Foundation Modelscited by 0
  4. 2026Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMscited by 0