# X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions

arXiv: https://arxiv.org/abs/2405.19744 · HTML: https://mlsanity.com/paper/2405.19744/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 30 May 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [Multilingual Constituency Parsing with Self-Attention and Pre-Training](https://mlsanity.com/paper/1812.11760/) (2018)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer](https://mlsanity.com/paper/2305.14857/) (2023)
- [Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Exploration](https://mlsanity.com/paper/2310.09168/) (2023)

## Built on by (most cited first, 18 of 18)

- [Unlocking the Potential of Model Merging for Low-Resource Languages](https://mlsanity.com/paper/2407.03994/) (2024)
- [Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque](https://mlsanity.com/paper/2506.07597/) (2025)
- [Enhancing Non-English Capabilities of English-Centric Large Language Models through Deep Supervision Fine-Tuning](https://mlsanity.com/paper/2503.01275/) (2025)
- [XIFBench: Evaluating Large Language Models on Multilingual Instruction Following](https://mlsanity.com/paper/2503.07539/) (2025)
- [1+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators?](https://mlsanity.com/paper/2406.14721/) (2024)
- [Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment](https://mlsanity.com/paper/2503.04647/) (2025)
- [Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh](https://mlsanity.com/paper/2502.13647/) (2025)
- [Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study](https://mlsanity.com/paper/2507.14304/) (2025)
- [A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunities](https://mlsanity.com/paper/2507.08425/) (2025)
- [Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs](https://mlsanity.com/paper/2607.22039/) (2026)
- [XL-Suite: Cross-Lingual Synthetic Training and Evaluation Data for Open-Ended Generation](https://mlsanity.com/paper/2503.22973/) (2025)
- [GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data](https://mlsanity.com/paper/2505.17082/) (2025)
- [CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languages](https://mlsanity.com/paper/2509.16914/) (2025)
- [LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgish](https://mlsanity.com/paper/2510.07074/) (2025)
- [Why Low-Resource NLP Needs More Than Cross-Lingual Transfer: Lessons Learned from Luxembourgish](https://mlsanity.com/paper/2605.10714/) (2026)
- [LLMs Get Smarter from Targeted Synthetic Multilingual Data](https://mlsanity.com/paper/2608.15964/) (2026)
- [EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languages](https://mlsanity.com/paper/2609.05043/) (2026)
- [PlurVA-LLM-2026 Shared Task Track-1: Pluralistic Value Alignment in LLMs via Multilingual Fine-Tuning and Threshold Calibration](https://mlsanity.com/paper/2609.32382/) (2026)

## Abstract (arXiv)

Large language models respond well in high-resource languages like English but struggle in low-resource languages. It may arise from the lack of high-quality instruction following data in these languages. Directly translating English samples into these languages can be a solution but unreliable, leading to responses with translation errors and lacking language-specific or cultural knowledge. To address this issue, we propose a novel method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages. Specifically, the language model first learns to generate appropriate English instructions according to the natural web texts in other languages as responses. The candidate cross-lingual instruction tuning samples are further refined and diversified. We have employed this method to build a large-scale cross-lingual instruction tuning dataset on 10 languages, namely X-Instruction. The instruction data built using our method incorporate more language-specific knowledge compared with the naive translation method. Experimental results have shown that the response quality of the model tuned on X-Instruction greatly exceeds the model distilled from a powerful teacher model, reaching or even surpassing the ones of ChatGPT. In addition, we find that models tuned on cross-lingual instruction following samples can follow the instruction in the output language without further tuning.