# Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque

arXiv: https://arxiv.org/abs/2506.07597 · HTML: https://mlsanity.com/paper/2506.07597/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 30 of 33)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [What Language Model to Train if You Have One Million GPU Hours?](https://mlsanity.com/paper/2210.15424/) (2022)
- [Multilingual Instruction Tuning With Just a Pinch of Multilinguality](https://mlsanity.com/paper/2401.01854/) (2024)
- [LLMs Are Few-Shot In-Context Low-Resource Language Learners](https://mlsanity.com/paper/2403.16512/) (2024)
- [A New Massive Multilingual Dataset for High-Performance Language Technologies](https://mlsanity.com/paper/2403.14009/) (2024)
- [FinGPT: Large Generative Models for a Small Language](https://mlsanity.com/paper/2311.05640/) (2023)
- [Latxa: An Open Language Model and Evaluation Suite for Basque](https://mlsanity.com/paper/2403.20266/) (2024)
- [Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus](https://mlsanity.com/paper/2410.14815/) (2024)
- [BertaQA: How Much Do Language Models Know About Local Culture?](https://mlsanity.com/paper/2406.07302/) (2024)
- [X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions](https://mlsanity.com/paper/2405.19744/) (2024)
- [UCCIX: Irish-eXcellence Large Language Model](https://mlsanity.com/paper/2405.13010/) (2024)

## Built on by (most cited first, 12 of 12)

- [Refusal Steering: Fine-grained Control over LLM Refusal Behaviour for Sensitive Topics](https://mlsanity.com/paper/2512.16602/) (2025)
- [Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates](https://mlsanity.com/paper/2512.04844/) (2025)
- [Small LLMs for Medical NLP: a Systematic Analysis of Few-Shot, Constraint Decoding, Fine-Tuning and Continual Pre-Training in Italian](https://mlsanity.com/paper/2602.17475/) (2026)
- [Physical Commonsense Reasoning for Lower-Resourced Languages and Dialects: a Study on Basque](https://mlsanity.com/paper/2602.14812/) (2026)
- [EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training](https://mlsanity.com/paper/2603.02041/) (2026)
- [Multimodal Large Language Models for Low-Resource Languages: A Case Study for Basque](https://mlsanity.com/paper/2511.09396/) (2025)
- [Automatic Essay Scoring and Feedback Generation in Basque Language Learning](https://mlsanity.com/paper/2512.08713/) (2025)
- [Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages](https://mlsanity.com/paper/2512.08777/) (2025)
- [SemBench: A Universal Semantic Framework for LLM Evaluation](https://mlsanity.com/paper/2603.11687/) (2026)
- [A Catalog of Basque Dialectal Resources: Online Collections and Standard-to-Dialectal Adaptations](https://mlsanity.com/paper/2603.25189/) (2026)
- [Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights](https://mlsanity.com/paper/2603.28263/) (2026)
- [Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study](https://mlsanity.com/paper/2605.28710/) (2026)

## Abstract (arXiv)

Instructing language models with user intent requires large instruction datasets, which are only available for a limited set of languages. In this paper, we explore alternatives to conventional instruction adaptation pipelines in low-resource scenarios. We assume a realistic scenario for low-resource languages, where only the following are available: corpora in the target language, existing open-weight multilingual base and instructed backbone LLMs, and synthetically generated instructions sampled from the instructed backbone. We present a comprehensive set of experiments for Basque that systematically study different combinations of these components evaluated on benchmarks and human preferences from 1,680 participants. Our conclusions show that target language corpora are essential, with synthetic instructions yielding robust models, and, most importantly, that using as backbone an instruction-tuned model outperforms using a base non-instructed model. Scaling up to Llama 3.1 Instruct 70B as backbone, our model comes near frontier models of much larger sizes for Basque, without using any Basque instructions. We release code, models, instruction datasets, and human preferences to support full reproducibility in future research on low-resource language adaptation. https://github.com/hitz-zentroa/latxa-instruct