# LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language

arXiv: https://arxiv.org/abs/2405.07745 · HTML: https://mlsanity.com/paper/2405.07745/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 60 references that are in the map.
- First posted on arXiv 13 May 2024.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Continual Pre-Training of Large Language Models: How to (re)warm your model?](https://mlsanity.com/paper/2308.04014/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1907.05774/) (2019)
- [Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs](https://mlsanity.com/paper/2305.16339/) (2023)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [ELLE: Efficient Lifelong Pre-training for Emerging Data](https://mlsanity.com/paper/2203.06311/) (2022)
- [CoEdIT: Text Editing by Task-Specific Instruction Tuning](https://mlsanity.com/paper/2305.09857/) (2023)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [Fine-tuning Large Language Models for Domain-specific Machine Translation](https://mlsanity.com/paper/2402.15061/) (2024)
- [RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization](https://mlsanity.com/paper/2401.14280/) (2024)
- [Cabrita: closing the gap for foreign languages](https://mlsanity.com/paper/2308.11878/) (2023)
- [Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2404.04042/) (2024)

## Built on by (most cited first, 6 of 6)

- [Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus](https://mlsanity.com/paper/2410.14815/) (2024)
- [SinLlama -- A Large Language Model for Sinhala](https://mlsanity.com/paper/2508.09115/) (2025)
- [Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review](https://mlsanity.com/paper/2505.04531/) (2025)
- [Adapting Chat Language Models Using Only Target Unlabeled Language Data](https://mlsanity.com/paper/2412.11704/) (2024)
- [Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning](https://mlsanity.com/paper/2505.09738/) (2025)
- [Cross-Lingual Transfer and Parameter-Efficient Adaptation in the Turkic Language Family: A Theoretical Framework for Low-Resource Language Models](https://mlsanity.com/paper/2604.06202/) (2026)

## Abstract (arXiv)

Despite advancements in English-dominant generative large language models, further development is needed for low-resource languages to enhance global accessibility. The primary methods for representing these languages are monolingual and multilingual pretraining. Monolingual pretraining is expensive due to hardware requirements, and multilingual models often have uneven performance across languages. This study explores an alternative solution by adapting large language models, primarily trained on English, to low-resource languages. We assess various strategies, including continual training, instruction fine-tuning, task-specific fine-tuning, and vocabulary extension. The results show that continual training improves language comprehension, as reflected in perplexity scores, and task-specific tuning generally enhances performance of downstream tasks. However, extending the vocabulary shows no substantial benefits. Additionally, while larger models improve task performance with few-shot tuning, multilingual models perform worse than their monolingual counterparts when adapted.