# Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation

arXiv: https://arxiv.org/abs/2412.13375 · HTML: https://mlsanity.com/paper/2412.13375/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [Extrapolating Large Language Models to Non-English by Aligning Languages](https://mlsanity.com/paper/2308.04948/) (2023)
- [Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation](https://mlsanity.com/paper/2205.12647/) (2022)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Mitigating Catastrophic Forgetting in Language Transfer via Model Merging](https://mlsanity.com/paper/2407.08699/) (2024)
- [Efficiently Adapting Pretrained Language Models To New Languages](https://mlsanity.com/paper/2311.05741/) (2023)
- [Romanization-based Large-scale Adaptation of Multilingual Language Models](https://mlsanity.com/paper/2304.08865/) (2023)
- [Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation](https://mlsanity.com/paper/2304.07854/) (2023)
- [TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes](https://mlsanity.com/paper/2311.10797/) (2023)
- [MIZAN: A Large Persian-English Parallel Corpus](https://mlsanity.com/paper/1801.02107/) (2018)
- [Leveraging ParsBERT and Pretrained mT5 for Persian Abstractive Text Summarization](https://mlsanity.com/paper/2012.11204/) (2020)
- [Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT](https://mlsanity.com/paper/2404.02403/) (2024)
- [PersianMind: A Cross-Lingual Persian-English Large Language Model](https://mlsanity.com/paper/2401.06466/) (2024)
- [LSCP: Enhanced Large Scale Colloquial Persian Language Understanding](https://mlsanity.com/paper/2003.06499/) (2020)
- [Bilingual Adaptation of Monolingual Foundation Models](https://mlsanity.com/paper/2407.12869/) (2024)
- [Extracting an English-Persian Parallel Corpus from Comparable Corpora](https://mlsanity.com/paper/1711.00681/) (2017)

## Built on by (most cited first, 3 of 3)

- [Improving Multilingual Capabilities with Cultural and Local Knowledge in Large Language Models While Enhancing Native Performance](https://mlsanity.com/paper/2504.09753/) (2025)
- [NagaNLP: Bootstrapping NLP for Low-Resource Nagamese Creole with Human-in-the-Loop Synthetic Data](https://mlsanity.com/paper/2512.12537/) (2025)
- [Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages](https://mlsanity.com/paper/2506.19187/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have made great progress in classification and text generation tasks. However, they are mainly trained on English data and often struggle with low-resource languages. In this study, we explore adding a new language, i.e., Persian, to Llama (a model with a limited understanding of Persian) using parameter-efficient fine-tuning. We employ a multi-stage approach involving pretraining on monolingual Persian data, aligning representations through bilingual pretraining and instruction datasets, and instruction-tuning with task-specific datasets. We evaluate the model's performance at each stage on generation and classification tasks. Our findings suggest that incorporating the Persian language, through bilingual data alignment, can enhance classification accuracy for Persian tasks, with no adverse impact and sometimes even improvements on English tasks. Additionally, the results highlight the model's initial strength as a critical factor when working with limited training data, with cross-lingual alignment offering minimal benefits for the low-resource language. Knowledge transfer from English to Persian has a marginal effect, primarily benefiting simple classification tasks.