mlsanityData to 7 Oct 2026

Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code

arXiv 2404.00399 ↗First posted 30 Mar 2024cs.CL cs.AI cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Pretrained language models are an integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim to democratize access to pretrained models for collaborative community development. Despite these efforts, such models encounter challenges such as limited multilingual capabilities, risks of catastrophic forgetting during continual pretraining, and the high costs of training models from scratch, alongside the need to align with AI safety standards and regulatory frameworks. This paper presents Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. Continually pretrained from StarCoderPlus on 435B additional tokens, Aurora-M surpasses 2T tokens in total training token count. It is the first open-source multilingual model fine-tuned on human-reviewed safety instructions, thus aligning its development not only with conventional red-teaming considerations, but also with the specific concerns articulated in the Biden-Harris Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. We evaluate Aurora-M across a wide range of tasks and languages, showcasing its robustness against catastrophic forgetting and its superior performance in multilingual settings, particularly in safety evaluations. We open-source Aurora-M and its variants to encourage responsible open-source development of large language models at https://huggingface.co/aurora-m.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 65 of its 100 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

65 of 100 references.

References · 65

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
38 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  9. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  10. 2017Mixed Precision Trainingcited by 1,135
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  12. 2022Red Teaming Language Models with Language Modelscited by 983
  13. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  14. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  15. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  16. 2022Galactica: A Large Language Model for Sciencecited by 736
  17. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  18. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  19. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  20. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  21. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  22. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  23. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  24. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  25. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  26. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  27. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  28. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  29. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  30. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  31. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  32. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  33. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  34. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
  35. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  36. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  37. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  38. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 3.

10 papers.

  1. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  2. 2024STAR: SocioTechnical Approach to Red Teaming Language Modelscited by 13
  3. 2024Generative AI and Teachers -- For Us or Against Us? A Case Studycited by 6
  4. 2024A Survey of Large Language Models for European Languagescited by 6
  5. 2025MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sourcescited by 3
  6. 2025The Structural Safety Generalization Problemcited by 2
  7. 2025Debiasing Multilingual LLMs in Cross-lingual Latent Spacecited by 2
  8. 2026The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Modelscited by 1
  9. 2026No Safe Dose: How Training Data Drives Unsafe Image Generationcited by 0
  10. 2026When Vocabulary Comprehension Fails Clinical Reasoning: Evaluating Therapy Bots' Safety Risks for Generation Alphacited by 0