# Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh

arXiv: https://arxiv.org/abs/2502.13647 · HTML: https://mlsanity.com/paper/2502.13647/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning](https://mlsanity.com/paper/2305.13627/) (2023)
- [X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions](https://mlsanity.com/paper/2405.19744/) (2024)
- [CIDAR: Culturally Relevant Instruction Dataset For Arabic](https://mlsanity.com/paper/2402.03177/) (2024)
- [KazNERD: Kazakh Named Entity Recognition Dataset](https://mlsanity.com/paper/2111.13419/) (2021)
- [Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Finance](https://mlsanity.com/paper/2410.01109/) (2024)
- [KazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Dataset](https://mlsanity.com/paper/2104.08459/) (2021)
- [MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions](https://mlsanity.com/paper/2409.12958/) (2024)
- [KazQAD: Kazakh Open-Domain Question Answering Dataset](https://mlsanity.com/paper/2404.04487/) (2024)
- [KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes](https://mlsanity.com/paper/2403.19335/) (2024)

## Built on by (most cited first, 4 of 4)

- [Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting](https://mlsanity.com/paper/2503.01493/) (2025)
- [LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgish](https://mlsanity.com/paper/2510.07074/) (2025)
- [Qomhra: A Bilingual Irish and English Large Language Model](https://mlsanity.com/paper/2510.17652/) (2025)
- [From National Curricula to Cultural Awareness: Constructing Open-Ended Culture-Specific Question Answering Dataset](https://mlsanity.com/paper/2601.04632/) (2026)

## Abstract (arXiv)

Instruction tuning in low-resource languages remains underexplored due to limited text data, particularly in government and cultural domains. To address this, we introduce and open-source a large-scale (10,600 samples) instruction-following (IFT) dataset, covering key institutional and cultural knowledge relevant to Kazakhstan. Our dataset enhances LLMs' understanding of procedural, legal, and structural governance topics. We employ LLM-assisted data generation, comparing open-weight and closed-weight models for dataset construction, and select GPT-4o as the backbone. Each entity of our dataset undergoes full manual verification to ensure high quality. We also show that fine-tuning Qwen, Falcon, and Gemma on our dataset leads to consistent performance improvements in both multiple-choice and generative tasks, demonstrating the potential of LLM-assisted instruction tuning for low-resource languages.