mlsanityData to 7 Oct 2026

Bilingual Adaptation of Monolingual Foundation Models

arXiv 2407.12869 ↗First posted 13 Jul 2024cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

We present an efficient method for adapting a monolingual Large Language Model (LLM) to another language, addressing challenges of catastrophic forgetting and tokenizer limitations. We focus this study on adapting Llama 2 to Arabic. Our two-stage approach begins with expanding the vocabulary and training only the embeddings matrix, followed by full model continual pre-training on a bilingual corpus. By continually pre-training on a mix of Arabic and English corpora, the model retains its proficiency in English while acquiring capabilities in Arabic. Our approach results in significant improvements in Arabic and slight enhancements in English, demonstrating cost-effective cross-lingual transfer. We perform ablations on embedding initialization techniques, data mix ratios, and learning rates and release a detailed training recipe. To demonstrate generalizability of this approach we also adapted Llama 3 8B to Arabic and Llama 2 13B to Hindi.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 13 Jul 2024.

Builds on

30 of 41 references.

References · 30

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
18 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  2. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  3. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  4. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  5. 2024Large Language Models: A Surveycited by 506
  6. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  7. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  8. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  9. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  10. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  11. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  12. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  13. 2020EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answeringcited by 60
  14. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  15. 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
  16. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  17. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  18. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2025Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-trainingcited by 4
  2. 2024Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptationcited by 3
  3. 2025Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Settingcited by 3
  4. 2025Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavariancited by 3
  5. 2025Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languagescited by 0